Python Metaprogramming: Functions, Flask and Google Cloud Functions
Everything in Python is an object. And that includes functions. Let’s see what I learned while I was trying to work with Google cloud functions with python r...
Posts by Category
- python 10
- AWS 5
- HBase 5
- Linux 4
- public speaking 4
- FPGA 3
- Zynq 2
- HDP 2
- Networking 2
- linux 2
- aws 1
- golang 1
python
Debugging a Running Python Process
Only if it were as easy as installing debug symbols, attach the process with gdb and py-bt! So we have a python agent, which distributes files, running acros...
PyCon19 India: Let’s Hunt a Memory Leak
We faced a memory leak in production and I wrote about it in this blog post. A while back, I somewhere came across the open Call for Proposals for Pycon Indi...
Curious Case of Python 2 and Integers
In Detecting Memory Leak in Python, scenarios were shown where python does not release memory when we created a huge list and then explicitly deleted it. The...
SRECon19 Asia: Let’s Build a Distributed File System
In the first post on this blog, I wrote about a tiny distributed filesystem I made in python for educational purpose. This year, I had a chance to use it in ...
Serverless Meets CI/CD
I have been attending LSPE [Large Scale Production Engineering] Meetup for last two years. And for the last one, I decided to give it back to the community. ...
Detecting Memory Leak in Python
In production, a memory leak will not always bubble up. And there could be multiple reasons behind it. You may not be getting enough traffic. Frequent deploy...
TCP Fast Open: In Action with Python
Recently I was revisiting concepts of TCP protocol and that reminded me that there was also a thing called TCP Fast Open. Digging further on the same reveale...
Writing Simple WebSocket Server in Python: PyWSocket
Echo websocket server implemented by hand on raw TCP Sockets.
Simple Distributed File System in Python : PyDFS
I was reading on HDFS (Hadoop’s distributed file system) and it’s internals. How does it store data. What is reading path. What is writing path. How does rep...
AWS
Encrypt Existing AWS RDS : The GDPR Series
Last, and the most dreaded, comes the MySQL RDS. The most critical part of infrastructure which is responsible for auth, new sign-ups and other user related ...
Encrypt Existing S3 Bucket : The GDPR Series
So next in line was S3 bucket. This too did not have encryption enabled, ie: data encryption at rest.
Encrypting Existing AWS EBS : The GDPR Series
It’s been some time since The GDPR has kicked in. And like every other ops person, I too had to work for compliance related tasks. Three major tasks that I t...
Setting up Inter Region AWS VPC Peering and Latency Tests
Most of our infrastructure and client facing services are in us-east-1 and we have lots of users connecting from different parts of the world including India...
Resize EBS Root Volume of CentOS 6 AMI
So the other day I had to create a CentOS 6 AMI for HDP installation as it had Hue package available only for CentOS 6. I launched an instance with EBS attac...
HBase
Upgrading Apache Phoenix in HDP Cluster
About new Hadoop cluster we set up, the phoenix version bundled with HDP distribution(4.7) had some bugs which would make it impossible to use to run BI quer...
HBase YouAreDeadException: Dead RegionServer due to GC Pause
So the CDH Cluster was replaced by HDP Cluster and everything was going smooth for time being. Until the time when I started getting a dead RegionServer. Fre...
Migrating OpenTSDB to Another HBase Cluster
As a part of migration from CDH cluster to HDP cluster, we also had to migrate OpenTSDB which was running on CDH cluster. There are many methods to copy/tran...
Stuff You Can Do While Tuning HBase
So you are setting up HBase! Congratulations!
HBase Benchmarking
Currently I am working with new setup of Apache HBase cluster to query data using Phoenix on top of HDP Distribution. After setting up cluster, the values f...
Linux
How to reconfigure Zynq-PL on-the-go?
You would have wondered if i’s possible to reconfigure the PL part without any interruption while PS is running Linux. Well, i’s possible and as simple as,
PYNQ Linux on ZedBoard
Hi There!
Resize EBS Root Volume of CentOS 6 AMI
So the other day I had to create a CentOS 6 AMI for HDP installation as it had Hue package available only for CentOS 6. I launched an instance with EBS attac...
Debugging Stuck Process in Linux
The other day I faced a problem with monitoring setup and I found that the WebUI is not responding. I SSHed into server and checked if process is running. It...
public speaking
SRECon’21: Leveraging ML to Detect Application HotSpots [@scale, of Course!]
Long pandemic no see! I submitted two proposals for SRECon 20 Asia which was supposed to happen in Australia. One talk got selected. But instead of the confe...
PyCon19 India: Let’s Hunt a Memory Leak
We faced a memory leak in production and I wrote about it in this blog post. A while back, I somewhere came across the open Call for Proposals for Pycon Indi...
SRECon19 Asia: Let’s Build a Distributed File System
In the first post on this blog, I wrote about a tiny distributed filesystem I made in python for educational purpose. This year, I had a chance to use it in ...
Serverless Meets CI/CD
I have been attending LSPE [Large Scale Production Engineering] Meetup for last two years. And for the last one, I decided to give it back to the community. ...
FPGA
How to reconfigure Zynq-PL on-the-go?
You would have wondered if i’s possible to reconfigure the PL part without any interruption while PS is running Linux. Well, i’s possible and as simple as,
PYNQ Linux on ZedBoard
Hi There!
Configure PS of PYNQ to work with SDK
Hi, If you’re an FPGA fan or someone who’s got PYNQ board for fun, you might be having a hard time making it run Vivado SDK projects. That’s because, the PY...
Zynq
How to reconfigure Zynq-PL on-the-go?
You would have wondered if i’s possible to reconfigure the PL part without any interruption while PS is running Linux. Well, i’s possible and as simple as,
PYNQ Linux on ZedBoard
Hi There!
HDP
Upgrading Apache Phoenix in HDP Cluster
About new Hadoop cluster we set up, the phoenix version bundled with HDP distribution(4.7) had some bugs which would make it impossible to use to run BI quer...
HBase YouAreDeadException: Dead RegionServer due to GC Pause
So the CDH Cluster was replaced by HDP Cluster and everything was going smooth for time being. Until the time when I started getting a dead RegionServer. Fre...
Networking
Setting up Inter Region AWS VPC Peering and Latency Tests
Most of our infrastructure and client facing services are in us-east-1 and we have lots of users connecting from different parts of the world including India...
TCP Fast Open: In Action with Python
Recently I was revisiting concepts of TCP protocol and that reminded me that there was also a thing called TCP Fast Open. Digging further on the same reveale...
linux
Debugging a Running Python Process
Only if it were as easy as installing debug symbols, attach the process with gdb and py-bt! So we have a python agent, which distributes files, running acros...
Site Wide Memory Leak: An On-Call Story
This happened a while back, sometime in the year of 2017. I was on-call for the week and it was the weekend. Usually, things are quiet over the weekends but ...
aws
Serverless Meets CI/CD
I have been attending LSPE [Large Scale Production Engineering] Meetup for last two years. And for the last one, I decided to give it back to the community. ...
golang
Golang Http Client and Compression
I had a very (seemingly) simple task. Verify my golang http client, talking with an ElasticSearch cluster, is compressing data on wire. Because in trials, th...