Debugging a Running Python Process

Only if it were as easy as installing debug symbols, attach the process with gdb and py-bt! So we have a python agent, which distributes files, running across the fleet. And on some random hosts, it went haywire! On those set of hosts, the process was using 100% of CPU and not doing anything meaningful work. Restarting the process fixes the problem. I had worked on debugging a stuck process, but this was the opposite....

December 18, 2019 · 7 min · Sanket

Site Wide Memory Leak: An On-Call Story

This happened a while back, sometime in the year of 2017. I was on-call for the week and it was the weekend. Usually, things are quiet over the weekends but not that weekend. Pages started coming frequently affecting different hosts. The alert was titled WARNING: Memory usage is more than 80% And that was just not one or set of hosts. This started coming from random hosts from across the infrastructure....

August 14, 2019 · 4 min · Sanket

How to reconfigure Zynq-PL on-the-go?

You would have wondered if i’s possible to reconfigure the PL part without any interruption while PS is running Linux. Well, i’s possible and as simple as, echo '0' > /sys/devices/soc0/amba/f8007000.devcfg/is_partial_bitstream #echo '1' for partial bitstreams cat whatever_the_bit_file_name_is.bit > /dev/xdevcfg Yeah, tha’s it! Make sure you’re running it as root. Don’t have a nice Linux running on ZedBoard yet? have a look at PYNQ Linux on ZedBoard

May 7, 2017 · 1 min · parth

PYNQ Linux on ZedBoard

Hi There! The PYNQ Linux is a fun, easy and maker-friendly Ubuntu 15.04 rootfs. It comes bundled with the PYNQ-Z1 board, and the official documentations doesn’t even utter a word on how to build or port this image on any other Zynq. Maybe because it’s too obvious how to do so. What you need to run Linux on any ARM board? BOOT image (BOOT.bin) kernel image (uImage) devicetree blob (devicetree.dtb) rootfs What we need to worry about?...

May 7, 2017 · 5 min · parth

Resize EBS Root Volume of CentOS 6 AMI

So the other day I had to create a CentOS 6 AMI for HDP installation as it had Hue package available only for CentOS 6. I launched an instance with EBS attached of 10 GB with CentOS 6. Went on to create AMI out of it with EBS size of 100GB. These all went good and I proceed with launching instances for HDP cluster (12 was the number of instances). Everything went good and installation was complete....

February 28, 2017 · 2 min · Sanket

Debugging Stuck Process in Linux

The other day I faced a problem with monitoring setup and I found that the WebUI is not responding. I SSHed into server and checked if process is running. It was. Checked if port was open. It was. So as it happened, the process was running and listening on port but it was stuck somewhere and it was not accepting connection. So there it was, a running stuck process. Now I could simply have restarted the stuck process but that wouldn’t tell me what actually happened and where it was stuck....

February 6, 2017 · 3 min · Sanket