Friday, August 31, 2012

Starting and Stopping Fusion Applications the Right Way

Overview

Starting and stopping Fusion Applications is a complex task that involves invoking commands for multiple components (including WebLogic Domains, OPMN-Managed Instances, Database instances, 3rd Party software, etc) in multiple hosts. This process may take several minutes and ensuring that it is done completely and correctly is essential for regular maintence activities (patching, backup, etc.) as well as full operation of the Fusion Applications environment.

Aside from the tools that are part of the technology components in Fusion Apps (e.g. opmn, WebLogic domain scripts, node manager, etc.), Fusion Applications ship with a tool called fastartstop which makes the lifecycle processes a lot simpler by enabling an administrator to start/stop/bounce multiple Fusion WebLogic Domains running on different servers, all with a single command on a single host. While this essential tool does most of the work involved in bringing up and down the Fusion Apps environment, it does not do all, so this document aims to explain all tasks involved in starting and stopping a Fusion Apps environment completely.

This post focuses on non-highly available environments (non-HA). For HA environments, you should perform similar tasks on the additional hosts and may use a tool such as srvctl to start the nodes on the RAC database.
 

Components

The tables below list the components that make up a complete Fusion Application install:
 

Identity Management

Component Name Notes Install Types
IDM Database
  • 1 or 2 databases (OID and OIM) depending on the install type. Can be installed on a single Oracle_Home or different ones
IDM EDG for FA
OID
  • 1 or 2 instances (Policy and Identity) depending on the install type (for non-HA environments)
  • OPMN-managed
IDM EDG for FA
OVD
  • 1 instance (for non-HA environments)
  • OPMN-managed
IDM EDG for FA
OIF
  • Not mandatory
  • 1 instance (for non-HA environments)
  • OPMN-managed
IDM EDG for FA
IDM WLS Domains
  • 1 or 2 domains (for Split-Domain topology)
IDM EDG for FA
IDM Node Managers
  • 1 on each host running IDM Domains
IDM EDG for FA
IDM OHS
  • 1 instance (for non-HA environments)
  • OPMN-managed
IDM EDG for FA

Fusion Applications

Component Name Notes Install Types
FA Database
  • 1 database
All
FA OHS
  • 1 instance (for non-HA environments)
  • OPMN-managed
All
BI Instance
  • 1 instance (for non-HA environments)
  • OPMN-managed
All
GOP
  • 1 instance (for non-HA environments)
  • OPMN-managed
SCM only
IIR
  • 1 instance (for non-HA environments)
CRM only
FA WLS Domains
  • 1 domain for each Product Family (up to 9): CommonDomain, CRMDomain, HCMDomain, FinancialDomain, SCMDomain, ICDomain, ProcurementDomain, ProjectsDomain, BIDomain
  • Domains installed depend on the offering selected during Provisioning
  • Can be started/stopped/bounced with a single command using the fastartstop tool
All
FA Node Manager
  • 1 on each FA server with WLS Domains
All

 

Starting Fusion Applications

The table below lists the tasks to be performed when starting Fusion Applications, in order of execution. Note that the sample code is provided as an example and will most likely be different depending on how your Fusion Applications environment was installed.
The following environment variables are used:
  • ORACLE_HOME: used for starting up Oracle database instances. It points at the Oracle Database home and varies depending on the database.
     
  • ORACLE_SID: used for starting up Oracle database instances. It varies depending on the database.
     
  • IDM_MW_HOME: Middleware home where Identity Management is installed. Depending on the install topology chosen during install, there may be more than one Middleware Home.
     
  • IDM_INSTANCE_HOME: Directory where AS Instances for OID, OVD, OHS are installed. Depending on the topology chosen during install, AS Instances may not all be located in the same directory.
     
  • FA_BASE: Directory where Fusion Applications are installed. Normally contains the "fusionapps" and "instance" directories (with installation defaults).
Component Sample Startup Code
Start Identity Management
Database instance(s) and DB Listener(s)
export ORACLE_HOME=<oracle_home>
export ORACLE_SID=<oracle_sid>
$ORACLE_HOME/bin/lsnrctl start
$ORACLE_HOME/bin/sqlplus / as sysdba
> startup
Node Manager(s)
$IDM_MW_HOME/wlserver_10.3/server/bin/startNodeManager &
disown 

OID
$IDM_INSTANCE_HOME/oid_inst1/bin/opmnctl startall

OVD
$IDM_INSTANCE_HOME/ovd_inst1/bin/opmnctl startall
OIF (if present)
$IDM_INSTANCE_HOME/oif_inst1/bin/opmnctl startall
WebLogic AdminServer(s)
$IDM_MW_HOME/oracle_common/common/bin/wlst.sh
> nmConnect('nm_user','nm_password','nm_server','nm_port', 'IDMDomain','IDMDomain_path');
> nmStart('AdminServer');
SOA
$IDM_MW_HOME/oracle_common/common/bin/wlst.sh
> nmConnect('nm_user','nm_password','nm_server','nm_port', 'IDMDomain','IDMDomain_path');
> nmStart('soa_server1');
* If already connected, simply run nmStart
OIM
$IDM_MW_HOME/oracle_common/common/bin/wlst.sh
> nmConnect('nm_user','nm_password','nm_server','nm_port', 'IDMDomain','IDMDomain_path');
> nmStart('oim_server1');
* If already connected, simply run nmStart
OAM
$IDM_MW_HOME/oracle_common/common/bin/wlst.sh
> nmConnect('nm_user','nm_password','nm_server','nm_port', 'IDMDomain','IDMDomain_path');
> nmStart('oam_server1');
* If already connected, simply run nmStart
ODSM
$IDM_MW_HOME/oracle_common/common/bin/wlst.sh
> nmConnect('nm_user','nm_password','nm_server','nm_port', 'IDMDomain','IDMDomain_path');
> nmStart('wls_ods1');
> exit() 
* If already connected, simply run nmStart
IDM OHS
$IDM_INSTANCE_HOME/oif_inst1/bin/opmnctl startall
Start Fusion Applications
Fusion Apps Database Instance and Listener
export ORACLE_HOME=
export ORACLE_SID=
$ORACLE_HOME/bin/lsnrctl start
$ORACLE_HOME/bin/sqlplus / as sysdba
> startup
Node Manager(s) on each FA server
$FA_BASE/fusionapps/wlserver_10.3/common/bin/nodemanager/<host_name>startNodeManagerWrapper.sh &
disown  
*Always start the FA Node Manager using the command above, don't use the regular startNodeManager
Fusion Applications Domains
$FA_BASE/fusionapps/applications/lcm/ad/bin/fastartstop.sh
-Start
-all
-username weblogic_fa
-appbase $FA_BASE/fusionapps/applications
-startAdminServer true
BI Instance
$FA_BASE/instance/BIInstance/bin/opmnctl startall
GOP (if present)
$FA_BASE/instance/gop_1/bin/opmnctl startall
IIR (if present)
$FA_BASE/InformaticaIR/bin/setfusionEnv.sh
$FA_BASE/InformaticaIR/bin/liup
$FA_BASE/InformaticaIR/bin/idsup
FA OHS
$FA_BASE/instance/CommonDomain_webtier/bin/opmnctl startall

 

Stopping Fusion Applications

The table below lists the tasks to be performed when stopping Fusion Applications, in order of execution. Note that the sample code is provided as an example only and will most likely be different depending on how your Fusion Applications environment was installed.
Component Sample Startup Code
Stop Fusion Applications
FA OHS
$FA_BASE/instance/CommonDomain_webtier/bin/opmnctl stopall
BI Instance
$FA_BASE/instance/BIInstance/bin/opmnctl stopall
GOP (if present)
$FA_BASE/instance/gop_1/bin/opmnctl stopall
IIR (if present)
$FA_BASE/InformaticaIR/bin/setfusionEnv.sh
$FA_BASE/InformaticaIR/bin/idsdown
$FA_BASE/InformaticaIR/bin/lidown
Fusion Applications Domains
$FA_BASE/fusionapps/applications/lcm/ad/bin/fastartstop.sh
-Stop
-all
-username weblogic_fa
-appbase $FA_BASE/fusionapps/applications
Fusion Apps Database Instance and Listener
$ORACLE_HOME/bin/lsnrctl stop
$ORACLE_HOME/bin/sqlplus / as sysdba
> shutdown immediate
> exit
Node Manager(s) on each FA server
ps -ef | grep weblogic.NodeManager
kill <pid_identified_above>
* There is no command or script to stop the Node Manager, so the process must be killed manually. To determine the process id, use the ps command above.
Stop Identity Management
IDM OHS
$IDM_INSTANCE_HOME/ohs_inst1/bin/opmnctl startall
SOA
$IDM_MW_HOME/oracle_common/common/bin/wlst.sh
> nmConnect('nm_user','nm_password','nm_server','nm_port', 'IDMDomain','IDMDomain_path');
> nmKill ('soa_server1');
* If already connected, simply run nmKill
* nmKill may require that the StopScriptEnabled property be set to true in the nodemanager.properties file (located in $MW_HOME/wlserver_10.3/common/nodemanager)
OIM
$IDM_MW_HOME/oracle_common/common/bin/wlst.sh
> nmConnect('nm_user','nm_password','nm_server','nm_port', 'IDMDomain','IDMDomain_path');
> nmKill ('oim_server1');
* If already connected, simply run nmKill
* nmKill may require that the StopScriptEnabled property be set to true in the nodemanager.properties file (located in $MW_HOME/wlserver_10.3/common/nodemanager)
OAM
$IDM_MW_HOME/oracle_common/common/bin/wlst.sh
> nmConnect('nm_user','nm_password','nm_server','nm_port', 'IDMDomain','IDMDomain_path');
> nmKill ('oam_server1');
* If already connected, simply run nmKill
* nmKill may require that the StopScriptEnabled property be set to true in the nodemanager.properties file (located in $MW_HOME/wlserver_10.3/common/nodemanager)
ODSM
$IDM_MW_HOME/oracle_common/common/bin/wlst.sh
> nmConnect('nm_user','nm_password','nm_server','nm_port', 'IDMDomain','IDMDomain_path');
> nmKill ('wls_ods1');
* If already connected, simply run nmKill
* nmKill may require that the StopScriptEnabled property be set to true in the nodemanager.properties file (located in $MW_HOME/wlserver_10.3/common/nodemanager)
WebLogic AdminServer(s)
$IDM_MW_HOME/oracle_common/common/bin/wlst.sh
> nmConnect('nm_user','nm_password','nm_server','nm_port', 'IDMDomain','IDMDomain_path');
> nmKill ('AdminServer');
> exit()
* If already connected, simply run nmKill
* nmKill may require that the StopScriptEnabled property be set to true in the nodemanager.properties file (located in $MW_HOME/wlserver_10.3/common/nodemanager)
OID
$IDM_INSTANCE_HOME/oid_inst1/bin/opmnctl stopall
OVD
$IDM_INSTANCE_HOME/ovd_inst1/bin/opmnctl stopall
OIF (if present)
$IDM_INSTANCE_HOME/oif_inst1/bin/opmnctl stopall
Database Instance(s) and DB Listener(s)
export ORACLE_HOME=<oracle_home>
export ORACLE_SID=<oracle_sid>
$ORACLE_HOME/bin/lsnrctl stop 
$ORACLE_HOME/bin/sqlplus / as sysdba
> shutdown immediate
> exit
Node Manager(s)
ps -ef | grep weblogic.NodeManager
kill <pid_identified_above>
* There is no command or script to stop the Node Manager, so the process must be killed manually. To determine the process id, use the ps command above.

 

15 comments:

  1. User's suddenly can't start FA when running the fastartstop script. Identity management is up and running. Any ideas on what would cause this?


    While trying to start up Fusion Applications, we're getting the following error:

    Error in initializing Credential Store
    [2012-09-05T13:40:44.044--6:00] [fastartstop] [ERROR:1] [SSCSF002] [CSF] [oracle.apps.startstop.util.CsfUtil: getCredential.115] [tid:10] Error in fetching Credential from CSF Store
    [2012-09-05T13:40:44.044--6:00] [fastartstop] [ERROR:1] [SSCSF003] [CSF] [oracle.apps.startstop.util.CsfUtil: getUserName.139] [tid:10] Error in CSF module, a required key was not found
    [2012-09-05T13:40:44.044--6:00] [fastartstop] [ERROR:1] [SSCSF002] [CSF] [oracle.apps.startstop.util.CsfUtil: getCredential.115] [tid:10] Error in fetching Credential from CSF Store
    [2012-09-05T13:40:44.044--6:00] [fastartstop] [ERROR:1] [SSCSF003] [CSF] [oracle.apps.startstop.util.CsfUtil: getPassword.159] [tid:10] Error in CSF module, a required key was not found
    [2012-09-05T13:40:44.044--6:00] [fastartstop] [ERROR:1] [SSUTIL019] [UTL] [oracle.apps.startstop.util.FSUtil: hookInitialContext.414] [tid:10] Error while getting DB connection
    [2012-09-05T13:40:44.044--6:00] [fastartstop] [ERROR:1] [] [UTL] [oracle.apps.startstop.util.FSUtil: hookInitialContext.414] [tid:10] SSUTIL019
    java.lang.NullPointerException
    at java.lang.String.(String.java:177)
    at oracle.apps.startstop.util.FSUtil.hookInitialContext(FSUtil.java:393)
    at oracle.apps.startstop.util.FSUtil.getAllDomainsFromFS(FSUtil.java:113)
    at oracle.apps.startstop.util.SSUtils.checkNodeManagerOnAllAdminHosts(SSUtils.java:292)
    at oracle.apps.startstop.invoke.InvocationMgrImpl.invoke(InvocationMgrImpl.java:218)
    at oracle.apps.startstop.cli.FAEnv.runStartStop(FAEnv.java:106)
    at oracle.apps.startstop.cli.FAEnv.main(FAEnv.java:167)

    ReplyDelete
    Replies
    1. This error is due to the password expired in the idm check the managed server's in IDM if you restart it ti will go to the admin state as some applications in the managed server will not start because if the expired password reset the database schema password the same error i got and reset the oid and oam database schema and password's in the error it shows that the error While getting the DBconnection

      Delete
  2. Great Article.. One question, shouldn't we also need to set (export WLST_PROPERTIES="-Dweblogic.security.SSL.trustedCAKeyStore=/u01/app/oracle/fa/products/fusionapps/wlserver_10.3/server/lib/fusion_trust.jks") property value before starting node manager (startNodeManagerWrapper.sh)?

    ReplyDelete
    Replies
    1. Thanks Kalpit. The Fusion Apps documentation doesn't mention this property when starting the node manager. The fusion_trust.jks store is already defined for each domain in the config.xml file, and the node manager by default uses the keystore _fusion_identity.jks, as defined in nodemanager.properties. Have you seen any situation where this was needed?

      Delete
  3. This comment has been removed by the author.

    ReplyDelete
  4. Hi ,

    You 've explained Oracle Fusion in easy to understand format for beginner like me

    Can you please suggest how to Install Financials suite of Oracle Fusion Applications... what are the sequence of steps if we Installing it on RHEL 5 server.

    Do we need to Install Oracle Database OR EBS R12 vision Database in order to Install to Financials suite of Oracle Fusion Applications

    Appreciate your reply if you could Enlightened a bit more




    Thanks & Regards
    MZ

    ReplyDelete
  5. Hi
    at a high level you need to install 2 databases (1 for Identity Management, and 1 for Fusion Applications - note these are not the EBS R12 databases).
    Then you must install the prerequisite identity management stack (this includes OIM, OAM, OVD, OID, etc). The doc explaining how to do that for version 11.1.4 is here:
    http://docs.oracle.com/cd/E28271_01/fusionapps.1111/e21032/toc.htm

    the equivalent doc for Fusion apps for 11.1.4 is here:
    http://docs.oracle.com/cd/E28271_01/fusionapps.1111/e27364/toc.htm

    I would suggest reading through these as a starting point.

    ReplyDelete
    Replies
    1. Hi Jennifer Briscoe

      Hope you 'll be in the best of Health.

      Thanks for the reply.

      I do 've some doubts over Installing The Financials suite of Oracle Fusion Applications on RHEL 5 server with RAM 16 GB.

      as my Understanding of Fusion Apps is really very Limited As Oracle eBusiness suite and Fusion Applications are totally different product suites.

      As I want to install one product (The Financials suite of Oracle Fusion Applications) in an Fusion Apps suite, DO I 've to install total Fusion applications suite ?

      Can I Exclude the Identity management Component Like SSO, OID, OAM while Installing Just The Financials suite of Oracle Fusion Applications On a single node Linux server ??

      Please Correct I am assuming that Financials suitu of Oracle Fusion Applications can be Installed Without Installing the Oracle Database Or EBS R12 vision Instance ?

      And If yes could you please share the Sequence of steps that need to be followed in order to install Just The Financials suite of Oracle Fusion Applications on RHEL 5 server.

      I would be Highly Obliged...

      Thanks & Regards


      Delete
  6. Mazhar
    With respect to your question on whether or not Fusion Financials can be installed onto a server with only 16GB of RAM, the answer is no. You will need to work with your sales representative to get your specific scenario properly sized by it is not uncommon to need 96GB of RAM or more and 500 GB of shared storage to start with. You specific requirements will depend on what you are trying to accomplish, meaning how many concurrent users, etc.

    Fusion Applications offers you the choice of which products you'd like installed during the provisioning process. You simply select the ones you want. However, there are some inter-dependencies between offerings. For example, even if you only want to install Financials a portion of HCM will be installed as User Management (create users, delete users, etc) is done through modules in HCM. Only those modules needed from HCM will be installed however, not the whole of HCM.

    No you cannot exclude the Identity Management components. All authentication and authorization is done through IDM.
    You can see in the document I referenced already that is it listed as a prerequisite.
    http://docs.oracle.com/cd/E28271_01/fusionapps.1111/e27364/ref_topologies_fn.htm#CHEGEJCD

    Fusion Applications requires a database as does the IDM stack. Again you do not need an EBS R12 instance, but you will need two separate databases. One for Fusion Applications and one for the IDM components.

    The sequence of steps to install the IDM components can be found here:
    http://docs.oracle.com/cd/E28271_01/fusionapps.1111/e21032/toc.htm

    The sequence of steps to install Fusion Applications (specifically Financials) can be found here:
    http://docs.oracle.com/cd/E28271_01/fusionapps.1111/e27364/toc.htm

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
    2. Hi Jennifer,

      Thanks for you Elaborative & Kind response. You are a Wonderful Person.
      Feel Indebted to you.

      *Could you please suggest which media pack for Oracle Fusion Apps should i download from edelivery out of
      following for Installing Fusion Applications (specifically Financials suite)*



      Select Description Release Part Number Updated # Parts / Size

      1. Oracle Fusion Applications 11g Release 4 (11.1.4) Media Pack for Linux x86-64 11.1.4.0.0 B67987-01 MAY-09-2012 22 /69G

      2. Oracle Fusion Applications 11g Release 4 (11.1.4) (with NLS Supplement) Media Pack for Linux x86-64 11.1.4.0.0 B68296-01 JUN-04-2012 33 /72G.

      3. Oracle Fusion Applications 11g RUP2 (11.1.3) Media Pack for Linux x86-64 11.1.3.0.0 B67014-01 MAR-30-2012 21 /65G

      4. Oracle Fusion Applications 11g RUP2 (11.1.3) (with NLS Supplement) Media Pack for Linux x86-64 11.1.3.0.0 B67223-01 APR-10-2012 32 /67G

      5. Oracle Fusion Applications 11g RUP1 (with NLS Supplement) Media Pack for Linux x86-64 11.1.2.0.0 B66413-02 MAR-23-2012 35 /
      65G
      6. Oracle Fusion Applications 11g Media Pack for Linux x86-64 n/a B63756-04 DEC-02-2011 24/63G

      Here is the Brief Info about the server.
      Root# uname
      x86_64

      cat /proc/meminfo
      MemTotal: 132275988 kB
      MemFree: 129592020 kB

      df -J [KH
      Filesystem Size Used Avail Use% Mounted on
      /dev/mapper/_root
      105G 1.2G 98G 2% /
      /dev/mapper/

      36G 185M 34G 1% /tmp
      /dev/mapper/
      21G 181M 20G 1% /home
      /dev/mapper/

      521G 208M 494G 1% /data
      /dev/sda 104M 17M 82M 17% /boot
      tmpfs 537M 0 537M 0% /dev/shm

      Is it Ok if we download the Software in /data dir. which has about 500GB space & than unzip in the same Directory ?


      Thanks & Regards
      MZ

      Delete
    3. Really Appreciate if you could tell what are the rpms (RPM LINUX PACKAGES) that 'll be required to for Installing Fusion Apps on Linux 64 bit (x86_64) Server. ?



      Thanks & Kind regards
      MZ

      Delete
  7. Could u please suggest anything as we r Getting This Strange error when we exceute the /runInstaller which is under /apps/oracle/installers/faprov/Disk1/ Dire
    .when we execute the command i.e ./runInstaller it check the Cpu speed temp space....etc & Than it ask or prompt for the jdk location... ... as soon as we specified the jdk location .. before opening the Installer Window....The terminal window get closed down automatically.

    Also xclock is working fine for root

    The jdk location which we provided to the prompt

    [root@SST~]# echo $JAVA_HOME
    /apps/oracle/jdk6/

    The server is Linux 64 bit & RAM is 132 GB & Harddisk space is 800 GB Also we 've installed the required RPMs & settup the Kernel parameters & rebooted the server as well

    we tried executing the runinstaller first from fusionapps User than after 2 hours min from root .... but got the same result.

    I am unable to figure it out .. Appreciate if you could suggest anything....

    Also if You Could please tell If it is Possible to Install Oracle Fusion Apps (Specifically Financial suite)on a single Linux Server which has around 132 GB of RAM & around 800 GB disk space. ?! There is No Oracle DB previously installed on this server. & we 've Downloaded the Oracle Fusion Applications 11g Release 4 (11.1.4) Media Pack for Linux x86-64.

    --
    Also if you Could please tell If it is possible to Install Oracle Fusion Apps (Specifically Financial suite)on a single Linux Server which has around 132 GB of RAM & around 800 GB disk space. ?! There is No Oracle DB previously installed on this server. & we 've Downloaded the Oracle Fusion Applications 11g Release 4 (11.1.4) Media Pack for Linux x86-64.


    Eagerly Awaiting your response


    Thanks & regards
    MZ

    ReplyDelete
  8. I am doing Oracle Fusion Applications 11g Installation while Running Fusion Applications Repository Creation Utility (Apps RCU) after the Tablespaces creation phase

    I am getting the following error while creating the required Schemas which is the last steps of rcu.

    ora-24010: Queue fusion.acr_ads_ims_queue does not exist
    ora-06512 at SYS.DBMS_AQADM LINE 270
    ORA-6512 AT LINE 98

    Can we Ignore the Error.

    Here is the content of fusionapps_postimport_latest.sql which is the Error file under/oracle/fusionapps/installers/apps_rcu/lrcu/integration/fusionapps/sql Direct.
    132 FA_DBINSTALL_CREATE_TRIG('logon_trigger_for_fusion after logon on
    133 fusion_runtime.schema
    134 ','begin
    135 execute immediate ''ALTER SESSION SET CURRENT_SCHEMA = FUSION'';
    136 end;
    137 ');
    138 FA_DBINSTALL_CREATE_TRIG('logon_trigger_for_fusion_bi after logon on fus ion_bi.schema
    139 ','begin
    140 execute immediate ''ALTER SESSION SET CURRENT_SCHEMA = FUSION'';
    141 end;
    142 ');
    143 END;
    144 DECLARE
    145 num_cpus NUMBER;
    146 BEGIN
    147 --################################# RECOMPILE SCHEMAS ################## ###############
    148 SELECT value INTO num_cpus FROM v$osstat WHERE stat_name = 'NUM_CPUS';
    149 SYS.UTL_RECOMP.RECOMP_PARALLEL(threads => num_cpus, schema => NULL);
    150 END;
    151 END;
    152 /

    Appreciate if anyone could suggest ... Can we Ignore the Error ?
    Thanks & regards
    MZ

    ReplyDelete
  9. Excellent work…Appreciate your work…thanks very much for sharing.

    Appreciate if u could suggest anything on the following Fusion Apps Install related issue.

    I am facing these rcu related error.
    RCU-6130:Action failed – RCU-6135 Error while trying to execute java action

    I encountered these Error while executing Fusion Applications Repository Creation Utility (Apps RCU)

    I am installing Oracle Fusion Applications 11g Release 4 (11.1.4) single node server.. The Linux server is 64 bit & RAM is 135 gb & disk space is 800 gb.

    There is No Oracle database previously installed on this server. & we ‘ve Downloaded the Oracle Fusion Applications 11g Release 4 (11.1.4) Media Pack for Linux x86-64.

    Please Advise..

    Regards
    Priya.

    ReplyDelete