SKA Project: The quest for real-time processing

for massive data streams




​​​"By leveraging our expertise in these areas, S[&]T is helping to shape the future of radio astronomy and unlock the secrets of the universe."

Background 

The Square Kilometer Array (SKA) is set to redefine our understanding of the universe. As the world's most powerful radio telescope, it will generate unprecedented amounts of data from its observation sites in Australia and South Africa.


Before the SKA takes centre stage, smaller precursor and pathfinder projects like MeerKAT and LOFAR are paving the way. These telescopes are generating valuable data to test and refine the algorithms that will be crucial for processing the SKA's massive data streams.


To harness the full potential of the SKA, we need innovative solutions to process its massive data streams in real-time. S[&]T is at the forefront of this challenge, developing cutting-edge algorithms and software tools to extract valuable scientific insights.

The Problem

Unlike traditional radio telescopes, the SKA will operate continuously, 24 hours a day, 7 days a week. This relentless data flow, estimated at around 10 terabits per second from each site, presents a formidable challenge: real-time processing on an unprecedented scale. This is extremely important since batch processing is not a solution for this case, as there is no possibility of storing this massive amount of data before processing, and by processing it in real-time, we may select interesting targets and discard the data with irrelevant information.. ​


The Solution

To tackle this challenge, the SKA project has made strategic decisions about its software infrastructure. Python, a versatile and widely used language, forms the core of the software stack, complemented by popular open-source libraries like NumPy and SciPy. For computationally intensive tasks, C++ is employed, leveraging its performance advantages.


The core tools will transform the signals captured by the antennas into useful information. Among the core tools two stood out as fundamental: WSClean and DP3. Both are used by the self-calibration pipeline, a set of processing steps coded in Python used to calibrate the data. Our team has contributed significantly to the development of core SKA software tools, including WSClean and DP3. These tools are essential for transforming raw radio signals into meaningful scientific data, such as images of distant galaxies and black holes.


To achieve the SKA goals, we work closely with the international astronomy community to ensure that our tools are robust, efficient, and accessible. By releasing our software as open-source, we foster collaboration and accelerate scientific discovery.


S[&]T's Crucial Role in SKA's Success

S[&]T played a pivotal role in the success of this project by providing the software engineering expertise that underpins many of the SKA’s achievements. One of our engineers was instrumental in developing and optimizing the software tools that process the SKA’s massive data streams. 


By contributing this key talent, we ensured the SKA team could meet its ambitious goals, tackling challenges such as real-time data processing and high-performance computation with confidence.

Our team's contributions to the SKA extend beyond software development. We are actively involved in:


​• Algorithm Development: Designing and implementing algorithms for data calibration, imaging, and analysis. 

​• Pipeline Development: Creating automated pipelines to process vast amounts of data efficiently. 

​• High-Performance Computing: Optimizing software for maximum performance on supercomputers. 
​
​• Data Science: Applying advanced data science techniques to extract valuable insights from the data.


For more information about our consulting expertise and how we support high-tech projects, visit High-​Tech Consulting.

Conclusion

Every day, we're pushing the boundaries of radio astronomy, bringing the SKA's ambitious goals closer to reality. Our team is actively involved in optimising tools like WSClean to improve real-time data processing, a critical capability for the SKA's success.

Beyond the SKA, our open-source software is already empowering astronomers worldwide. By sharing our tools and collaborating with the broader community, we're accelerating scientific discovery and solving complex challenges in radio astronomy.


Are you interested in knowing more? Contact our High-Tech Consulting team! 

​​Contact High-Tech Consulting