This is the pipeline we need to develop, Below is the deployment plan. We have accomplished few Stages / Tasks :
The deployment is on 11 FTP Servers :
Backup – FTP Server
2. Checkout – On any server where we have HDFS, Hbase & RabbitMQ – May be 115
3. Build the jar – On any server where we have HDFS, Hbase & RabbitMQ – May be 115
Config files need to be pushed to Git Repo – 11 Config files (paths can be found on confluence)
4. Deploying configuration files & other files – FTP Server
Need sudo su – access for devops_atm_prod to root, sbgetl, Eeauser, uccasetl – Capture all accounts for com.tmobile.cdr.filewatcher.FileWatcher ————————————————————————– Action Items
Go for root access. Request for it.
5. Creating Directory paths in the HDFS – On any server with HDFS – Ex : 115
6. Ensure HBase tables are created. – On any server with Hbase – Ex : 115
7. Stop File watcher to deploy new jar if its existing filewatcher – command is in Confluence page- FTP Server
Kill the process ID with “kill – pid”, How to find the process ID of correct file_watcher???
jps -m | grep fw_application.properties.sbgevent.ha (this will be changing, need to fetch from git commit PR)
8. Check if filewacher is stopped or not, check if file count has to be changed.
Should be with correct user
Should be in correct path
Instructions are in last column in Conflience :
• Login with sbgetl user
• path: /opt/deployments/sbgevent/filewatcher/sbgevent/
• kinit -kt /etc/security/keytabs/sbgetl.application.keytab sbgetl@HDPQUANTUMPROD.COM
9. Deploy Jar artifacts to 11 FTP servers – FTP Server – Is it replacing existing jar
Ex :
nohup java -Xmx8092m -Dname=FileWatcher-scef -cp /opt/deployments/scef/filewatcher/scef/:/opt/deployments/scef/filewatcher/bin/file-watcher-1.3.9-jar-with-dependencies.jar:`hbase classpath` com.tmobile.cdr.filewatcher.FileWatcher /opt/deployments/scef/filewatcher/prop/fw_application.properties.scef.ha &
Here file-watcher-1.3.9-jar-with-dependencies.jar has to be set as Variable. If version no is provide we don’t need to have build stage running else run build and pick the jar name from the deployment.
10. Based on the config files, need to create queues in RabbitMQ – need Rest API – What is the host ? – On any server which has access to RabbitMQ – May by 115 – Shayam to help with the POC
11. Start file watcher to test new jar – need command – FTP Server
Start command is in 3rd Column
Validation –
12. Check if file watcher started or not
Check if files are getting processed and the count is changing.
Questions :
1. Getting all the required accesses, mainly root aswell on all 11 FTP hosts.
2. Create RabbitMQ queues, hbase tables , local files & directories and HDFS Directories. Code template is in place for hbase tables, local files & directories and HDFS Directory creation.
2.1 We need to find RestAPI command for RMQ queue management.
Inputs :
User secret test in Jenkins but file should be rid of password.
3. Planning to run filewatcher with latest Jar Version, is that fine ?
All file watcher should be running on same version. Any challenges ??
4. Config files has RabbitMQ logins, so can it be pushed to Git ? Should this be encrypted and then decrypted using secret test ?
Inputs:
Need to encrypt when pushed to Git.
Go / No-Go with CI/CD ?
