# Step-by-Step Guide: Running Kafka Producers and Consumers on Oracle Cloud VPS with Databricks

In a recent Big Data Analytics lab assignment, we were tasked with ingesting (consuming) data into Databricks from an external system. To tackle this challenge, I explored setting up data producers on a cloud server—leveraging the fact that these servers already have public IPs, making external communication much easier. After diving deep into configurations and testing, I put together this concise guide to help others set up a similar pipeline efficiently.

## Allow External Connections on Your VPS

First, we need to enable external connections on our VPS. I’ve already covered that in a separate guide, which you can refer to here.

## Setting Up Your Server

1. Login to your Linux Server using SSH
    
2. Install Java
    
    ```bash
    sudo apt install default-jdk
    ```
    
3. Download Kafka
    
    ```bash
    wget https://archive.apache.org/dist/kafka/3.2.0/kafka_2.12-3.2.0.tgz
    tar -xzf kafka_2.12-3.2.0.tgz
    sudo mv kafka_2.12-3.4.0 /usr/local/kafka
    ```
    
4. Create Sytem file units
    
    ```bash
    nano /etc/systemd/system/zookeeper.service
    ```
    
    ```plaintext
    [Unit]
    Description=Apache Zookeeper server
    Documentation=http://zookeeper.apache.org
    Requires=network.target remote-fs.target
    After=network.target remote-fs.target
    
    [Service]
    Type=simple
    ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
    ExecStop=/usr/local/kafka/bin/zookeeper-server-stop.sh
    Restart=on-abnormal
    
    [Install]
    WantedBy=multi-user.target
    ```
    
    ```bash
    nano /etc/systemd/system/kafka.service
    ```
    
    ```plaintext
    [Unit]
    Description=Apache Kafka Server
    Documentation=http://kafka.apache.org/documentation.html
    Requires=zookeeper.service
    
    [Service]
    Type=simple
    Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-arm64"
    ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
    ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh
    
    [Install]
    WantedBy=multi-user.target
    ```
    
    Modify your java path in Environemt="JAVA\_HOME=<Your\_Path>“ as required. You can find it using
    
    ```bash
    readlink -f $(which java)
    ```
    
5. Modify server config
    
    ```bash
    nano /usr/local/kafka/config/server.properties
    ```
    
    Paste the following under Socket Server Settings.
    
    ```plaintext
    listeners=PLAINTEXT://0.0.0.0:9092,PLAINTEXT_EXTERNAL://0.0.0.0:9093
    advertised.listeners=PLAINTEXT://localhost:9092,PLAINTEXT_EXTERNAL://<YOUR_PUBLIC_IP>:9093
    listener.security.protocol.map=PLAINTEXT:PLAINTEXT,PLAINTEXT_EXTERNAL:PLAINTEXT
    ```
    
    replace &lt;YOUR\_PUBLIC\_IP&gt; with the public IP above.
    
6. Reload `systemd daemon`
    
    ```bash
    sudo systemctl daemon-reload
    ```
    
7. Start zookeeper & kafka server
    
    ```bash
    sudo systemctl start zookeeper
    sudo systemctl start kafka
    sudo systemctl status kafka
    ```
    
8. Create a Topic
    
    ```bash
    usr/local/kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic myTopic
    ```
    

## On Databricks

Create a new notebook after initializing a cluster, create the following cells

```python
%sh
java -version
```

```python
%sh
wget https://archive.apache.org/dist/kafka/3.2.0/kafka_2.12-3.2.0.tgz
ls -ltr ./
```

```python
%sh
tar -xzf kafka_2.12-3.2.0.tgz
```

```python
%sh
cd kafka_2.12-3.2.0
bin/kafka-console-consumer.sh --topic myTopic --bootstrap-server <YOUR_PUBLIC_IP>:9093
```

replace &lt;YOUR\_PUBLIC\_IP&gt; with the public IP above.