2 Dec 2016

A year on my own…

Today I completed one year of being on my own.

In 2004, after graduating from college and doing brief experiment with a startup I came to Pune, looking for a job. I failed clearing the interviews of many MNCs. In the last two and a half years of my college I spent most of my time learning Linux and Computer networks. So, I decided to narrow down my search where there is requirement of some Linux Admin work and after giving my first interview, I got the job. Anyways coming forward, I do had some plans after leaving my full time job but didn’t know what coming next. As last time Linux helped again but this time in the form of  The Linux Foundation. I got an assignment from The Linux Foundation to build a self paced course on “Cloud Infrastructure Technologies”, which got launched in June’16.

Between Dec’15 – March’16 months I also gave free containers(Docker) workshops in Bangalore, Pune, Chennai, Hyderabad and Kolkatta. At the same time I gave some paid workshops as well.

Within a month after leaving my job I realised how easy it be an employee than your own boss. One has to manage his/her time, next month’s pay check, family, health and so on without loosing the cool. After going through on my own I started to respect entrepreneurs, self employed, my local vendors, auto rickshaw drivers etc more.

Less sleep, lot of work took a toll on my body and I got struck with Bell’s Palsy in Jan’16. It was very scary to go under MRI in the middle of nigh for the checkup. It took me few months to recover. This definitely effected my work and I had to re-organise myself. As everyone I thought of getting some interns and employees but that did not work either. I spent my time, energy and money to make them upto speed but did not succeed either. It was difficult to part ways with some of them but I am happy that we did that in good terms. It was a good experience which I think can come only with few mistake. I learnt one very important lesson “Never hire a full-time until I am 100% satisfied”.

The formal registration of the company did not happen until May 2016. It is one marathon task as well. Working with Charted Account to get all documentation is not fun but it has to be done anyway. For registration I had to also give a company name. I spent good amount of time think about it. I was looking for some inspiration/help which I got from Ranga Shankara, a theatre in Bangalore and very near to my office. On my way to back from daily visit to a coffee shop I saw the board for theatre festival “Youth Yuga”. In hindi “Yuga” means “Era”, which made me think that is an era of cloud computing. So why not name my company “CloudYuga”. I took the “.guru” domain as in next year or two I would focusing on trainings.

Till Dec’15 to July’16 we worked from Bangalore Alpha Lab and then moved to our own office. It was fun to see the our own office taking shape.

_20161130_081246

Throughout the year I was engaged with Docker Community, which helped me both professionally and psychologically. I became part of Docker Captain’s program, which was a good confidence booster. In Bangalore I organised Docker meetups, which kept me well connected with local community. On my own I attended different international conferences DevConf’16, DockerCon’16 and LinuxCon/ContainerCon’16 & spoke in two of them .

When I started last year I told my family that lets see for 6 months and if things does not work out then I can go back to full time again. Its been year and I have things in pipeline for next 6 months, which is good. My family supported me very well, specially my wife Kanika who also joined to work with me part time since last few months.

Till now my focus has been training than consulting, which I believe would continue for some time. I see good amount skill gap in adopting container technologies.. good for me !!. Over the year I got some corporate clients for trainings which I think would continue to grow. Here is the group photo of the container (Docker) training I delivered today in Pune.

DSC_0141

Some of learning from last year’s experience are:-

  • There are more good people in the world than we think. They are willing to help you.
  • Reach out to people with helping hand.
  • Give priority to health.
  • Be true to yourself and things would fall in place.

Its been a fun ride with lots of new experienced. Lets see how far we can go !!

21 Nov 2016

Docker Mentor Week 2016 – Bangalore

Docker organised Global Mentor Week around the world between 14-19th Nov’16. In Bangalore we scheduled it on 19th Nov at Microsoft office. We divided the group into two groups Beginner and Intermediate of 125 each. Out of which ~60 turned up for each session.

The content for the mentor week was shared by the Docker, which had hands-on exercises for basic and intermediate level.  Microsoft provided Azure pass for every participant so that we all can have same working environment. But during the workshop we faced some internet issue, so everyone could not hands-on on that day.  But with Azure pass and labs, participants can try out the hands-on later.

For the beginner level we had four mentors Sreenivas MakamBathrinath Raveendran, Ratheesh T M  and myself. Sreenivas did the heavy lifting by walking through the Docker basics and labs. He also shared the notes to try out Docker Mentor Week’s lab on Azure. We had the beginner session between ~9:40 AM to 12:145 PM.

After the lunch we did the intermediate training. As the internet was not working, we decided to share the concept of Container Orchestration in general and then with Docker Swarm we showcased a sample application. I used the content from my LinuxCon/ContainerCon workshop which I did last month in Berlin.

After that Ajeet showed how to prepare Docker Swarm cluster on Azure and run & scale the sample application.

Both sessions were very interactive. I hope more people would have joined from the confirmed list. I had to say no to few group members as they could not get the confirmed ticket.

Thanks to Pracheta, Usha, Surubhi  and Sudhir form Microsoft for helping us with the Venue and Azure Passes.

20 Nov 2016

Gluster Geo-replication Tools

A must have tools collection for Gluster Geo-replication users!

Currently this repository contains gluster-georep-setup and gluster-georep-status tools. More tools will be added in future. Let me know if you need any specific tool to manage Gluster Geo-replication.

gluster-georep-setup was previously called as georepsetup (Blog about georepsetup is here).

Installation

Install the tools collection using, pip install gluster-georep-tools. Binary packages are not yet available, hopefully I will work on the packaging in near future.

gluster-georep-status

Wrapper around Geo-rep status command and Volume info command to provide more features compared to Gluster CLI. This tool combines Geo-rep status and Volume info to get following advantageous.

  • Nodes will be displayed in the same order as in Volume info
  • Offline nodes are shown with "Offline" as status
  • Status output from different sessions are not mixed.
  • Filters are available(Ex: --with-status=active, --with-crawl-status=changelog, --with-status=faulty etc)
  • Shows summary of number of workers per status

Example output(Listing all the sessions, gluster-georep-status:

SESSION: gv1 ==> fvm1::gv2
+-----------------+--------+-----------------+------------+---------------------+------------+-----------------+-----------------------+
|      MASTER     | STATUS |   CRAWL STATUS  | SLAVE NODE |     LAST SYNCED     | CHKPT TIME | CHKPT COMPLETED | CHKPT COMPLETION TIME |
+-----------------+--------+-----------------+------------+---------------------+------------+-----------------+-----------------------+
| fvm1:/bricks/b1 | Active | Changelog Crawl |    fvm1    | 2016-11-14 08:34:40 |    N/A     |       N/A       |          N/A          |
| fvm1:/bricks/b2 | Active | Changelog Crawl |    fvm1    | 2016-11-14 08:32:21 |    N/A     |       N/A       |          N/A          |
+-----------------+--------+-----------------+------------+---------------------+------------+-----------------+-----------------------+
Active: 2 | Passive: 0 | Faulty: 0 | Created: 0 | Offline: 0 | Stopped: 0 | Initializing: 0 | Total: 2

SESSION: gv1 ==> geoaccount@fvm1::gv3
+-----------------+---------+--------------+------------+-------------+------------+-----------------+-----------------------+
|      MASTER     |  STATUS | CRAWL STATUS | SLAVE NODE | LAST SYNCED | CHKPT TIME | CHKPT COMPLETED | CHKPT COMPLETION TIME |
+-----------------+---------+--------------+------------+-------------+------------+-----------------+-----------------------+
| fvm1:/bricks/b1 | Stopped |     N/A      |    N/A     |     N/A     |    N/A     |       N/A       |          N/A          |
| fvm1:/bricks/b2 | Stopped |     N/A      |    N/A     |     N/A     |    N/A     |       N/A       |          N/A          |
+-----------------+---------+--------------+------------+-------------+------------+-----------------+-----------------------+
Active: 0 | Passive: 0 | Faulty: 0 | Created: 0 | Offline: 0 | Stopped: 2 | Initializing: 0 | Total: 2

gluster-georep-setup

In previous blog, we discussed about this tool. This tool simplifies the steps involved in Geo-replication setup. Now setting up Geo-replication is as easy as running one command. Yay!

Gluster Geo-rep Setup

Usage instructions of all the tools are available here

Let me know if these tools are useful.

1 Nov 2016

Jack of all trades: New Cisco UCS S-Series and Red Hat Storage

imagesToday, Cisco announced its new UCS S-Series storage-optimized server with the introduction of the UCS S3260, marking its entry into the emerging server market for data intensive workloads.

Red Hat and Cisco have worked together for a long time, including our collaboration on Red Hat OpenStack Platform.

Out with the old…

By jumping into the high-density storage-optimized server market, Cisco validates what we see as the continued movement to emerging software-defined, scale-out architectures for solutions like OpenStack and container-native storage and hyper-converged infrastructure.

With the ability to spread data across multiple servers, both Red Hat Ceph Storage and Red Hat Gluster Storage are helping to drive this trend. Open, software-defined storage enables enterprises to build an elastic cloud infrastructure for newer, data intensive workloads.

Ceph provides unified storage over a distributed object store (RADOS) as its core by providing unified block, object and file interfaces, while Gluster provides an elastic, scale out NAS file storage system.

As more organizations move to open source SDS from appliances / traditional SAN arrays, they often miss the recipes for a best practice deployment. Red Hat has worked with Cisco to produce reference design architectures to take the guess work out of configuring throughput-optimized, cost / capacity-optimized and emerging high IOPs performing clusters, including whitepapers for both Red Hat Ceph Storage and Red Hat Gluster Storage with Cisco’s previous generation of the S-Series, the C3160 high density rack server.

Open source drives storage innovation

Both Ceph and Gluster use community-powered innovation to accelerate their core feature sets faster than what is possible via a single proprietary vendor. Red Hat is a top contributor to both Ceph and Gluster upstream development, but several hardware, software and cloud service providers, including eBay, Yahoo!, CERN (Ceph) and Facebook (Gluster), all contribute to the code base. Cisco itself is a top-50 contributor to Ceph in terms of code commits.

Versatility

The Cisco UCS S-Series builds on the x86 storage-optimized server trend – but seemingly shuffles the deck with more of an enterprise spin via features such as dual-node servers, quadruple fans and power supplies, connected to Cisco UCS Fabric Interconnects.

One aspect of the new UCS S-Series design we are excited about is “versatility”. UCS offers common, consistent architecture for variety of IT needs that we expect may enable it to become a standard hardware building block for enterprise environments. S-Series includes features such as a modular chassis design, facilitating upgrades to new Intel chipsets including its disk expander module, providing the ability to swap out a server node for an additional 4 drives (increasing the raw capacity from 560 to 600 TB).

Cisco has also integrated networking fabric into its storage-optimized servers, making it easier to extend your interconnect as your cluster scales out. The S3260 offers dual 40GbE ports for each server node. As one moves to denser servers (with more than 24 drives) in Ceph configurations, the need for 40Gb Ethernet becomes greater. Enterprises can benefit from tightly-integrated fabric interconnect which translates to less latency, which is important for applications like video streaming.

A key piece is the UCS Manager configuration and handling tool which can simplify deployment. UCS Manager enables the creation of an initial configuration profile for storage, network, compute, etc. for the S3260, helping customers to more easily grow their Ceph environments by pushing out the profile to additional S3260s as they expand.

Combined with the Red Hat Storage ability to handle block, object and file access along with being flexible enough to handle throughput optimized, cost / capacity and high IOPS workloads, Cisco’s UCS S-Series may not just be a jack of all trades, but also a master of many.

Stay tuned for more upcoming joint solution papers from the Cisco UCS S3260 and Red Hat Ceph Storage teams. In the interim, learn more about the UCS S-Series at cisco.com/go/storage.

29 Oct 2016

[How to] GlusterFS Dynamic Volume Provisioner in Kubernetes (>= v1.4 ) / Openshift .

I am happy to share that Gluster Dynamic Volume Provisioner is available in kubernetes tree since 1.4 release !!

This is one of the feature which enables the User/Developer in Kubernetes/Openshift to have Persistent Volume Claim ( PVC ) request to be satisfied dynamically without an admin intervention. IMHO, it gives a nice user experience and its a good feature to have in container orchestrators. Till this feature came in, the persistent volumes in the store was created statically.

The static provisioning workflow looks like this:

static

Eventhough it was easy to perform the static provisioning of volumes, it has few limitations in my view.

*) The admin has to create the persistent volumes upfront and keep it in persistent store.

*) When a claim request comes to the controller, it check for the size of the request against the available PVs in the pool and if the ( available PV size >= size of the request ) it bind the claim.

The latter can lead to wastage of storage in most of the case.

These kind of limitations have been lifted with Dynamic provisioning. Now the admin define the storage classes and the user/developer request the persistent volumes using the storage class reference in the PVC request. The storage classes can pass the parameters of the plugin using the storage class key value pairs.

I have created the following diagram to simply the workflow of dynamic (Ex: GlusterFS) provisioning.

dyn_provisioner_latest

As you can see in the above diagram, the GlusterFS plugin in Kubernetes/Openshift make use of “Heketi” to provision GlusterFS volumes. If you want to know more about Heketi and how it can be used to manage gluster clusters, please refer this Wiki. In short, Heketi is a volume manager for GlusterFS clusters. It manage Gluster trusted pool and create volumes based on demand.

Lets start from Storage Class which allows us to do dynamic provisioning in kubernetes.

Here is an example of the storage class parameters:

apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
name: slow
provisioner: kubernetes.io/glusterfs
parameters:
resturl: “http://127.0.0.1:8081”
restuser: “admin”
secretNamespace: “default”
secretName: “heketi-secret”

Where

resturl : Gluster REST service/Heketi service url which provision gluster volumes on demand. The general format should be IPaddress:Port and this is a mandatory parameter for GlusterFS dynamic provisioner. If Heketi service is exposed as a routable service in openshift/kubernetes setup, this can have a format similar to heketi-storage-project.cloudapps.mystorage.com where the fqdn is a resolvable heketi service url.

restuser : Gluster REST service/Heketi user who has access to create volumes in the Gluster Trusted Pool.

secretNamespace + secretName : Identification of Secret instance that containes user password to use when talking to Gluster REST service. These parameters are optional, empty password will be used when both secretNamespace and secretName are omitted.

for more details on these parameters please refer github.com/kubernetes/kubernetes/tree/master/examples/experimental/persistent-volume-provisioning

To summarize, user/developer request the persistent storage using the claim and mention the storage class which need to be used/mapped with the claim. As soon as the claim request comes in, the GlusterFS plugin in kubernetes create a volume with the requested size and BIND the persistent volume to Claim. When there is a request to delete the claim, the subjected volume is deleted from the backend gluster trusted pool. The glusterfs plugin in kubernetes make use of ‘Heketi’ to provision a volume dynamically.

Here is the demo video of Dynamic GlusterFS provisioner in Kubernetes.

As always comments/suggestions/questions are welcome.. 🙂

Volgen-2.0

Designing and prototyping volgen for GD2

I’ve recently begun working on volgen for GD2. This post gives an overview of what I’ve been doing till now.

Some background first

One of the requirements for GD2 is to make it easier to modify the volume graphs, in particular make it easier to add new translators into the volume graph.

GlusterD right now has hardcoded graph topologies in its code. This makes it very hard to begin modify graphs. This made it hard for features like tiering to do their stuff.

For GD2, we are designing a more flexible volgen package, that makes it much easier to modify, extend and combine graphs. We’ve begun working on a prototype to explore how this design would work in real life.

Different approaches to volgen was discussed earlier this year in a meeting during DevConf. The approaches discussed are summarized in the “Flexible volgen” wiki. Of the discussed approaches, the “SystemD units” style approach was picked as the most suitable.

I’ve begun a prototype implementation of this approach at kshlm/glusterd2-volgen. I welcome comments and queries on the approach as well as the prototype.

The SystemD style volgen

NOTE: Most of the names, paths etc. used below are just placeholders. They will all be changed to more decsriptive terms before we reach the end.

More information on this can be found on the wiki, but the wiki might now be always up to date. So I’m providing a quick summary of the approach here.

This approach to volgen makes use of what I’m currently calling Nodefiles. These are analogus to the systemd unit files. A Nodefile describes a node which can appear in a volume graph. There are two types of nodes/nodefiles, Xlators and Targets

Xlators

Xlators nodefiles describe GlusterFS translators. Nodefiles are text files, which have to follow a defined format, that describe the particular translator. Examples of Nodefiles can be found at kshlm/glusterd2-volgen/examples/xlator. Note: These examples are just that; examples. They are not the final form.

In addition to describing a translator, Nodefiles specify the dependencies of the translator on other translator. Some of the dependecies currently available are, - Requires, this translator requires the specified translator also to be enabled - Conflicts, this translator cannot work with the specified translator - Before, this translator must appear before the other translator in the volume graph - After, this translator must appear after the specified translator in the volume graph - Parent, this translator can only have the specified translator as its parent - Child, this translator can only have the specified translator as its child(ren).

In the prototype, TOML is used as the description language. The Nodefile is described in code as a the Node struct in volgen/nodefile.go. Note: This is incomplete right now.

Each translator will provide its own nodefile, all of which will need to be added to a directory known to GD2 (let’s call it XLATORDIR for now), for eg. /usr/lib/glusterd/xlators. To make it easier for authors to create these files, we may provide tools to scrape information from the C code and build the file. But that’s a task for later.

GD2 will read the Xlator nodefiles from XLATORDIR at startup and build an internal translator map, and a transaltor options table.

Targets

Targets are nodes which describe graphs. You can have Targets for FUSE graph, brick graph, self-heal daemon etc.

On disc, Targets are directories which have Xlators linked into it. The linked in translators are the translators which will be present in the graph. The Targets decribe themselves with an info nodefile in the directory. Specific dependencies for the target can also be given in the nodefile. An example target for the brick graph can be found at kshlm/glusterd2-volgen/examples/brick.target. A Target struct is described in volgen/target.go.

Targets can include other targets among themselves. This will make it easier to describe complex graphs. For eg. a fuse.target for the FUSE graph can include a performance.targetand clients.target among other translators. The performance.target includes all known performance translators, and would provide an easy way to enable/disable performance translators. The clients.target would just include the cluster translators, ie the part of the client graph below and including DHT. This will make it easier to generate other graphs, like NFS for example, which would now need to only need to include the clients.target instead of the whole client graph.

Targets are the starting points of building an actual volume graph. Once a target is loaded into GD2, building a graph will be as simple as calling Target.BuildGraph(volinfo).

The BuildGraph function will first do a dependency resolution of all the translators/targets included in the target and build a intermediate graph. BuildGraph is called on any targets in this graph, and the returned graph is inserted at the targets location. Build graph will next fill in this intermediate graph with information from the volinfo, disable xlators if required and generate the actual volume graph. Calling graph.Write() will write out a volfile describing this graph.

The protoype contains a simple app which shows how to make use of this volgen packaged.

Things left to be done

  • I still need to design and implement the actual dependency resolver. I expect to take a significant amount of time for this. I’ll keep everyone updated on my progress on this.
  • I still have only a vague picture of how to handle assigning identities and options for client xlators, ie. how to get volgen to assign a client xlator to a brick.
  • There is a question of how a graph with complex interconnections, such as the one for server side replication, can be described with this. This is something I expect to have an answer as I proceed further with the prototype.

19 Oct 2016

Truncating open log files

There are a number of applications that open a log file and just keep writing to it. This is, of course, a problem when you rotate logs as the application just keeps writing to the renamed log instead of opening the new one.

In logrotate you can handle this with the copytruncate option. It will copy the contents of the log to a new file and truncate the old one. Your application will merrily continue appending to the newly 0-sized log.

If you want to do that manually, consider the truncate command:

truncate --size=0 /var/log/foo.log

GlusterFS 3.8.5 is ready for consumption

An other month, an other GlusterFS 3.8 update! We're committed to fix reported bugs in the 3.8 Long-Term-Maintenance version, with monthly releases. Here is glusterfs-3.8.5 for increased stability. Packages for different distributions should be landing shortly.

Release notes for Gluster 3.8.5

This is a bugfix release. The Release Notes for 3.8.0, 3.8.1, 3.8.2, 3.8.3 and 3.8.4contain a listing of all the new features that were added and bugs fixed in the GlusterFS 3.8 stable release.

Bugs addressed

A total of 27 patches have been merged, addressing 26 bugs:
  • #1373723: glusterd experiencing repeated connect/disconnect messages when shd is down
  • #1374135: Rebalance is not considering the brick sizes while fixing the layout
  • #1374280: rpc/xdr: generated files are filtered with a sed extended regex
  • #1374573: gluster fails to propagate permissions on the root of a gluster export when adding bricks
  • #1374580: Geo-rep worker Faulty with OSError: [Errno 21] Is a directory
  • #1374596: [geo-rep]: AttributeError: 'Popen' object has no attribute 'elines'
  • #1374610: geo-replication *changes.log does not respect the log-level configured
  • #1374627: Worker crashes with EINVAL errors
  • #1374632: [geo-replication]: geo-rep Status is not showing bricks from one of the nodes
  • #1374640: glusterfs: create a directory with 0464 mode return EIO error
  • #1375043: bug-963541.t spurious failure
  • #1375096: dht: Update stbuf from servers having layout
  • #1375098: Value of `replica.split-brain-status' attribute of a directory in metadata split-brain in a dist-rep volume reads that it is not in split-brain
  • #1375542: [geo-rep]: defunct tar process while using tar+ssh sync
  • #1375565: Detach tier commit is allowed when detach tier start goes into failed state
  • #1375959: Files not being opened with o_direct flag during random read operation (Glusterfs 3.8.2)
  • #1375990: Enable gfapi test cases in Gluster upstream regression
  • #1376385: /var/tmp/rpm-tmp.KPCugR: line 2: /bin/systemctl: No such file or directory
  • #1376390: Spurious regression in tests/basic/gfapi/bug1291259.t
  • #1377193: Poor smallfile read performance on Arbiter volume compared to Replica 3 volume
  • #1377290: The GlusterFS Callback RPC-calls always use RPC/XID 42
  • #1379216: rpc_clnt will sometimes not reconnect when using encryption
  • #1379284: warning messages seen in glusterd logs for each 'gluster volume status' command
  • #1379708: gfapi: Fix fd ref leaks
  • #1383694: GlusterFS fails to build on old Linux distros with linux/oom.h missing
  • #1383882: client ID should logged when SSL connection fails

18 Oct 2016

GlusterFS 3.7.16

GlusterFS 3.7.16 released (very very late)

GlusterFS 3.7.16 has been finally released, over 2 weeks late. The late release was due to a combination of the Gluster summit and me falling sick. This is a regular scheduled release for GlusterFS-3.7 and includes 20 bug fixes since 3.7.15. The release-notes can be read here.

The tarball can be downloaded from download.gluster.org.

The CentOS Storage SIG packages have been built and are available in the centos-gluster37-test repository. These should be available in the release repository soon.

Packages for other distros should become available soon. See the READMEs in the respective subdirs at download.gluster.org for more details on how to obtain them.

10 Oct 2016

LinuxCon ContainerCon 2016 – Berlin Germany

LinuxCon, ContainerCon 2016 in Berlin was very special for me, as this is the first time I represented my company CloudYuga as a speaker in an international conference. The event was scheduled between 4th to 7th Oct.’16 .

I reached there on 3rd Oct’16 and got a chance to meet Jim Zemlin Executive Director of  The Linux Foundation. On 4th, during the keynote Solomon Hykes, Founder and CTO of Docker open-sourced Infrakit, which is a toolkit for creating and managing declarative, self-healing infrastructure. It is currently used in Docker for AWS and other similar solution. Later in the day I attended sessions on Comparing Container Orchestration tools by fellow Docker Captain  Adrian Mouat, Cgroups session by Michael Kerrisk and Mircoservices vs Reverse-Proxy via Emile Vague. In the evening I attended the speaker and sponsor’s party at House of Weekend, Berlin.

On 5th I spent most of the time preparing for container orchestration workshop on 7th. Though I attended few sessions like Getting started with Mesos Marathon by Brenden Matthews,  OCI session by Qiang Huang, Docker Orchestration via Aaron Lehmann,  Phil Estes talk on container security and Migrating traditional & existing app on container by  Scott McCarty. In the evening I attended 25th Anniversary of Linux Celebration The Great Orangery at Charlottenburg Palace. There I got the chance to meet Dan Kohn, Executive Director of The Cloud Native Computing Foundation and discussed about the course I did for Linux Foundation, earlier this year.

On 6th attended after attending one or two sessions I joined fellow Docker Captains for Jerome Docker Swarm workshop. After the workshop Solomon joined us for Q &A. He shared an interesting perspective on Docker – Docker which has been there for quite some time and Docker which is now evolving …

After that we had Docker Captain’s party at some nearby restaurant.

On 7th I delivered the hands-on session on container orchestration, in which I covered Docker Swarm, Kubernetes and Mesos Marathon. For Docker Swarm and Marathon I gave instances on DigitalOcean to each participant. For Kubernetes I created a cluster on Google Cloud and had a namespace created for each participant. I heard some good feedback about the workshop after the session was over.

After the session I went to attend Docker Distributed Summit for rest of the day. The summit had very good technical content. All of the videos are recorded and would be shared soon. I could not attend the second day of Summit as I had come back on 8th.

As always this had been a great experience. Learned a lot, met many interesting people and made some new friends.

People have been asking me who is funding your foreign trips as I attended DevConf in Brno,CZ and DockerCon in Seattle earlier this year as well.  I have been funding my own trips, which I think as investments. I got few leads from this conference as well. Hope I am in right path by doing so. Let’s see if I would have another blog post for overseas conference or not.

25 Sep 2016

Effective Gluster Monitoring using Events APIs

Last week I got opportunity to present about Gluster Events APIs in Gluster meetup Bangalore.

Events APIs will be available with Gluster 3.9 release.(Release Candidate is available if anybody interested in testing)

Without Events APIs, one way to get status of Cluster is by calling Gluster status command/api in periodic intervals.

Below illustration shows calling status once every 10 seconds.

Get Cluster status without Events APIs

With Events APIs, setup a listener Webhook and register with Gluster using gluster-eventsapi webhook-add <URL>. Call Gluster status command whenever webhook receives an Event.(Check this blog to know Webhooks and Events APIs in detail)

Get Cluster status with Events APIs

As part of presentation, created some visualizations to show how these real time notifications can be used to refresh the UI automatically when Gluster cluster state changes.

Following gif shows the UI change immediately after creating a Gluster Volume.

Gluster Volume Create and Start

When a brick process is killed,

Brick Process Killed

Volume Stop,

Volume Stop

References:

  • Documentation for the Events APIs feature is available here.
  • Gif images created using byzanz tool.(Example: sleep 2; byzanz-record --duration=5 --x=500 --y=0 --width=1024 --height=800 volume_stop.gif
  • Illustrations are created using mypaint software and Wacom Tablet.
  • Dashboard prototype was created using Python Flask + Elm + Websockets

15 Sep 2016

GlusterFS 3.8.4 is available, Gluster users are advised to update

Even though the last release 3.8 was just two weeks ago, we're sticking to the release schedule and have 3.8.4 ready for all our current and future users. As with all updates, we advise users of previous versions to upgrade to the latest and greatest. Several bugs have been fixed, and upgrading is one way to prevent hitting known problems in future.

Release notes for Gluster 3.8.4

This is a bugfix release. The Release Notes for 3.8.0, 3.8.1, 3.8.2 and 3.8.3 contain a listing of all the new features that were added and bugs fixed in the GlusterFS 3.8 stable release.

Bugs addressed

A total of 23 patches have been merged, addressing 22 bugs:
  • #1332424: geo-rep: address potential leak of memory
  • #1357760: Geo-rep silently ignores config parser errors
  • #1366496: 1 mkdir generates tons of log messages from dht xlator
  • #1366746: EINVAL errors while aggregating the directory size by quotad
  • #1368841: Applications not calling glfs_h_poll_upcall() have upcall events cached for no use
  • #1368918: tests/bugs/cli/bug-1320388.t: Infrequent failures
  • #1368927: Error: quota context not set inode (gfid:nnn) [Invalid argument]
  • #1369042: thread CPU saturation limiting throughput on write workloads
  • #1369187: fix bug in protocol/client lookup callback
  • #1369328: [RFE] Add a count of snapshots associated with a volume to the output of the vol info command
  • #1369372: gluster snap status xml output shows incorrect details when the snapshots are in deactivated state
  • #1369517: rotated FUSE mount log is using to populate the information after log rotate.
  • #1369748: Memory leak with a replica 3 arbiter 1 configuration
  • #1370172: protocol/server: readlink rsp xdr failed while readlink got an error
  • #1370390: Locks xlators is leaking fdctx in pl_release()
  • #1371194: segment fault while join thread reaper_thr in fini()
  • #1371650: [Open SSL] : Unable to mount an SSL enabled volume via SMB v3/Ganesha v4
  • #1371912: gluster system:: uuid get hangs
  • #1372728: Node remains in stopped state in pcs status with "/usr/lib/ocf/resource.d/heartbeat/ganesha_mon: line 137: [: too many arguments ]" messages in logs.
  • #1373530: Minor improvements and cleanup for the build system
  • #1374290: "gluster vol status all clients --xml" doesn't generate xml if there is a failure in between
  • #1374565: [Bitrot]: Recovery fails of a corrupted hardlink (and the corresponding parent file) in a disperse volume

1 Sep 2016

GlusterFS 3.7.15

GlusterFS 3.7.15 released

GlusterFS 3.7.15 has been released. This is a regular scheduled release for GlusterFS-3.7 and includes 38 bug fixes since 3.7.14. The release-notes can be read here.

Downloads

The tarball can be downloaded from download.gluster.org.

Packages

Binary packages have been built are in the process of being made available as updates.

The CentOS Storage SIG packages have been built and will become available in the centos-gluster37-test repository (from the centos-release-gluster37 package) shortly. These will be made available in the release repository after some more testing.

Packages for Fedora 23 are queued for testing in Fedora Koji/Bodhi. They will appear first via dnf in the Updates-Testing repo, then in the Updates repo.

Packages for Fedora 24, 25, 26; Debian wheezy, jessie, and stretch, are available now on download.gluster.org.

Packages for Ubuntu Trusty, Wily, and Xenial are available now in Launchpad.

Packages for SuSE available now in the SuSE build system.

See the READMEs in the respective subdirs at download.gluster.org for more details on how to obtain them.

Next release

GlusterFS-3.7.16 will be the next release for GlusterFS-3.7, and is currently targetted for release on 30th September 2016. The tracker bug for GlusterFS-3.7.16 has been created. Bugs that need to be included in 3.7.16 need to be marked as dependencies of this bug.

23 Aug 2016

The out-of-order GlusterFS 3.8.3 release addresses a usability regression

On occasion the Gluster projects deems an out-of-order release the best approach to address a problem that got introduced with the last update. The 3.8.3 version is such a release, and we advise all users to upgrade to it, if possible skipping the 3.8.2 release. See the included release notes for more details. We're sorry for any inconvenience caused.

Release notes for Gluster 3.8.3

This is a bugfix release. The Release Notes for 3.8.0, 3.8.1 and 3.8.2 contain a listing of all the new features that were added and bugs fixed in the GlusterFS 3.8 stable release.

Out of Order release to address a severe usability regression

Due to a major regression that was not caught and reported by any of the testing that has been performed, this release is done outside of the normal schedule.
The main reason to release 3.8.3 earlier than planned is to fix bug 1366813:
On restarting GlusterD or rebooting a GlusterFS server, only the bricks of the first volume get started. The bricks of the remaining volumes are not started. This is a regression caused by a change in GlusterFS-3.8.2.
This regression breaks automatic start of volumes on rebooting servers, and leaves the volumes inoperable. GlusterFS volumes could be left in an inoperable state after upgrading to 3.8.2, as upgrading involves restarting GlusterD.
Users can forcefully start the remaining volumes, by doing running the gluster volume start <name> force command.

Bugs addressed

A total of 24 patches have been merged, addressing 21 bugs:
  • #1357767: Wrong XML output for Volume Options
  • #1362540: glfs_fini() crashes with SIGSEGV
  • #1364382: RFE:nfs-ganesha:prompt the nfs-ganesha disable cli to let user provide "yes or no" option
  • #1365734: Mem leak in meta_default_readv in meta xlators
  • #1365742: inode leak in brick process
  • #1365756: [SSL] : gluster v set help does not show ssl options
  • #1365821: IO ERROR when multiple graph switches
  • #1365864: gfapi: use const qualifier for glfs_*timens()
  • #1365879: [libgfchangelog]: If changelogs are not available for the requested time range, no proper error message
  • #1366281: glfs_truncate missing
  • #1366440: [AFR]: Files not available in the mount point after converting Distributed volume type to Replicated one.
  • #1366482: SAMBA-DHT : Crash seen while rename operations in cifs mount and windows access of share mount
  • #1366489: "heal info --xml" not showing the brick name of offline bricks.
  • #1366813: Second gluster volume is offline after daemon restart or server reboot
  • #1367272: [HC]: After bringing down and up of the bricks VM's are getting paused
  • #1367297: Error and warning messages related to xlator/features/snapview-client.so adding up to the client log on performing IO operations
  • #1367363: Log EEXIST errors at DEBUG level
  • #1368053: [geo-rep] Stopped geo-rep session gets started automatically once all the master nodes are upgraded
  • #1368423: core: use for makedev(3), major(3), minor(3)
  • #1368738: gfapi-trunc test shouldn't be .t

12 Aug 2016

The GlusterFS 3.8.2 bugfix release is available

Pretty much according to the release schedule, GlusterFS 3.8.2 has been released this week. Packages are available in the standard repositories, and moving from testing-status in different distributions to normal updates.

Release notes for Gluster 3.8.2

This is a bugfix release. The Release Notes for 3.8.0 and 3.8.1 contain a listing of all the new features that were added and bugs fixed in the GlusterFS 3.8 stable release.

Bugs addressed

A total of 54 patches have been merged, addressing 50 bugs:
  • #1339928: Misleading error message on rebalance start when one of the glusterd instance is down
  • #1346133: tiering : Multiple brick processes crashed on tiered volume while taking snapshots
  • #1351878: client ID should logged when SSL connection fails
  • #1352771: [DHT]: Rebalance info for remove brick operation is not showing after glusterd restart
  • #1352926: gluster volume status client" isn't showing any information when one of the nodes in a 3-way Distributed-Replicate volume is shut down
  • #1353814: Bricks are starting when server quorum not met.
  • #1354250: Gluster fuse client crashed generating core dump
  • #1354395: rpc-transport: compiler warning format string
  • #1354405: process glusterd set TCP_USER_TIMEOUT failed
  • #1354429: [Bitrot] Need a way to set scrub interval to a minute, for ease of testing
  • #1354499: service file is executable
  • #1355609: [granular entry sh] - Clean up (stale) directory indices in the event of an rm -rf and also in the normal flow while a brick is down
  • #1355610: Fix timing issue in tests/bugs/glusterd/bug-963541.t
  • #1355639: [Bitrot]: Scrub status- Certain fields continue to show previous run's details, even if the current run is in progress
  • #1356439: Upgrade from 3.7.8 to 3.8.1 doesn't regenerate the volfiles
  • #1357257: observing " Too many levels of symbolic links" after adding bricks and then issuing a replace brick
  • #1357773: [georep]: If a georep session is recreated the existing files which are deleted from slave doesn't get sync again from master
  • #1357834: Gluster/NFS does not accept dashes in hostnames in exports/netgroups files
  • #1357975: [Bitrot+Sharding] Scrub status shows incorrect values for 'files scrubbed' and 'files skipped'
  • #1358262: Trash translator fails to create 'internal_op' directory under already existing trash directory
  • #1358591: Fix spurious failure of tests/bugs/glusterd/bug-1111041.t
  • #1359020: [Bitrot]: Sticky bit files considered and skipped by the scrubber, instead of getting ignored.
  • #1359364: changelog/rpc: Memory leak- rpc_clnt_t object is never freed
  • #1359625: remove hardcoding in get_aux function
  • #1359654: Polling failure errors getting when volume is started&stopped with SSL enabled setup.
  • #1360122: Tiering related core observed with "uuid_is_null () message".
  • #1360138: [Stress/Scale] : I/O errors out from gNFS mount points during high load on an erasure coded volume,Logs flooded with Error messages.
  • #1360174: IO error seen with Rolling or non-disruptive upgrade of an distribute-disperse(EC) volume from 3.7.5 to 3.7.9
  • #1360556: afr coverity fixes
  • #1360573: Fix spurious failures in split-brain-favorite-child-policy.t
  • #1360574: multiple failures of tests/bugs/disperse/bug-1236065.t
  • #1360575: Fix spurious failures in ec.t
  • #1360576: [Disperse volume]: IO hang seen on mount with file ops
  • #1360579: tests: ./tests/bitrot/br-stub.t fails intermittently
  • #1360985: [SNAPSHOT]: The PID for snapd is displayed even after snapd process is killed.
  • #1361449: Direct io to sharded files fails when on zfs backend
  • #1361483: posix: leverage FALLOC_FL_ZERO_RANGE in zerofill fop
  • #1361665: Memory leak observed with upcall polling
  • #1362025: Add output option --xml to man page of gluster
  • #1362065: tests: ./tests/bitrot/bug-1244613.t fails intermittently
  • #1362069: [GSS] Rebalance crashed
  • #1362198: [tiering]: Files of size greater than that of high watermark level should not be promoted
  • #1363598: File not found errors during rpmbuild: /var/lib/glusterd/hooks/1/delete/post/S57glusterfind-delete-post.py{c,o}
  • #1364326: Spurious failure in tests/bugs/glusterd/bug-1089668.t
  • #1364329: Glusterd crashes upon receiving SIGUSR1
  • #1364365: Bricks doesn't come online after reboot [ Brick Full ]
  • #1364497: posix: honour fsync flags in posix_do_zerofill
  • #1365265: Glusterd not operational due to snapshot conflicting with nfs-ganesha export file in "/var/lib/glusterd/snaps"
  • #1365742: inode leak in brick process
  • #1365743: GlusterFS - Memory Leak - High Memory Utilization

2 Aug 2016

GlusterFS 3.7.14

GlusterFS 3.7.14 released

GlusterFS 3.7.14 has been released. This is a regular scheduled release for GlusterFS-3.7 and includes 26 bug fixes since 3.7.13. The release-notes can be read here.

Downloads

The tarball can be downloaded from download.gluster.org.

Packages

Binary packages have been built are in the process of being made available as updates.

The CentOS Storage SIG packages have been built and will become available in the centos-gluster37-test repository (from the centos-release-gluster37 package) shortly. These will be made available in the release repository after some more testing.

Packages for Fedora 23 are queued for testing in Fedora Koji/Bodhi. They will appear first via dnf in the Updates-Testing repo, then in the Updates repo.

Packages for Fedora 24, 25, 26; epel 5, 6, 7; debian wheezy, jessie, and stretch, are available now on download.gluster.org.

Packages for Ubuntu Trusty, Wily, and Xenial are available now in Launchpad.

Packages for SuSE SLES-12, OpenSuSE 13.1, and Leap42.1 are available now in the SuSE build system.

See the READMEs in the respective subdirs at download.gluster.org for more details on how to obtain them.

Next release

GlusterFS-3.7.15 will be the next release for GlusterFS-3.7, and is currently targetted for release on 30th August 2016. The tracker bug for GlusterFS-3.7.15 has been created. Bugs that need to be included in 3.7.15 need to be marked as dependencies of this bug.