In production, a memory leak will not always bubble up. And there could be multiple reasons behind it. You may not be getting enough traffic. Frequent deployments. No hard memory usage limit set. Or mix of them. The flask app we had to debug had same characteristics. It never had huge surge of traffic and there would be multiple deployments over week. Although it had cgroup memory usage limit, it had some room to grow and the leak never appeared....
Last, and the most dreaded, comes the MySQL RDS. The most critical part of infrastructure which is responsible for auth, new sign-ups and other user related activities. The task was dreaded because there is no way to enable encryption once RDS has been created and most ways, which we will discuss, incurred downtime. The Task Encrypt existing MySQL RDS, which is also multi AZ, with near-zero downtime. The RDS instance we had was a MySQL one which is also multi AZ....
So next in line was S3 bucket. This too did not have encryption enabled, ie: data encryption at rest. The Task Encrypt existing S3 bucket which contains user data with zero downtime. A word on encrypted S3 objects/buckets: By default there is no encryption involved when you create or put objects in an S3 bucket. However, you can enable default encryption on a bucket and any object put in the bucket will be encrypted by default....
It’s been some time since The GDPR has kicked in. And like every other ops person, I too had to work for compliance related tasks. Three major tasks that I took up, which involved mostly encrypting data at rest, were Encrypting existing EBS Encrypt S3 buckets Encrypt RDS In this GDPR series, I will be sharing my experiences and how did we went on with it. The Task Encrypt EBS that is currently in use by a MongoDB cluster....
Most of our infrastructure and client facing services are in us-east-1 and we have lots of users connecting from different parts of the world including India. Of course there was a significant latency involved when users connect to US from other part of the world. And we wanted to test that, if a user from India connects to Mumbai region(faster handshake) and then that region uses VPC peering to us-east-1 to talk to other services....
About new Hadoop cluster we set up, the phoenix version bundled with HDP distribution(4.7) had some bugs which would make it impossible to use to run BI queries. There was no way provided by HDP to upgrade phoenix as we were using the latest version. Looking around on the internet, I found that manually we can replace the related jars and bins to have a new version in place. So that’s what I tried....
Recently I was revisiting concepts of TCP protocol and that reminded me that there was also a thing called TCP Fast Open. Digging further on the same revealed a lot. We will briefly discuss how this enhancement works. What are the limitations. And later we will do the hands on and see how the TCP Fast Open drastically reduces the load time. What is TCP Fast Open? TCP Fast Open is an optimization over TCP which eliminates the need to wait for 3 way handshake before application can send data over it....
Echo websocket server implemented by hand on raw TCP Sockets. Journey to websocket was pretty long. I started with an idea to make an app which can play music in sync across the devices during college period. No wonder I couldn’t get through it. Later this year I stumbled upon this new thing called WebSockets and they were intriguing. I thought I could finish that app with websockets (and I did, with partial success)....
So the CDH Cluster was replaced by HDP Cluster and everything was going smooth for time being. Until the time when I started getting a dead RegionServer. Frequently. So a deep dive was needed to dig out what indeed was happening. And it turned out to be a long dive. The following was the logline: 2017-05-23 06:59:22,173 FATAL [regionserver/<hostname>/10.10.205.55:16020] regionserver.HRegionServer: ABORTING region server <hostname>,16020,1493962926376: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing<hostname>,16020,1493962926376 as dead server This alone did not tell much....
You would have wondered if i’s possible to reconfigure the PL part without any interruption while PS is running Linux. Well, i’s possible and as simple as, echo '0' > /sys/devices/soc0/amba/f8007000.devcfg/is_partial_bitstream #echo '1' for partial bitstreams cat whatever_the_bit_file_name_is.bit > /dev/xdevcfg Yeah, tha’s it! Make sure you’re running it as root. Don’t have a nice Linux running on ZedBoard yet? have a look at PYNQ Linux on ZedBoard