Blog moves to self-hosted web-site

Dear friends!

I’ve decided to extend my blog to a personal website. It’s new address is nikolay.ivankin.pro.

I’m going to continue posting new articles, but this is not all – now you can find a knowledge base, a little bit empty yet ;-) . But I’m sure it will fill up very fast: all you need is just asking questions, share your problems.

Please, don’t be shine and come, visit it.

To all my followers: WordPress.com will move your subscriptions to new site.

Hope to see you soon, Nikolay Ivankin.

Wierd Things with Oracle :-)

Few weeks ago one of my clients had met a really wierd thing: there were 2 logfiles with CURRENT status:

SQL> select GROUP#,SEQUENCE#,BYTES,MEMBERS,STATUS from v$log
  2  ;
    GROUP#  SEQUENCE#      BYTES    MEMBERS STATUS
     ———- ———- ———- ———- —————-
         1       5651   52428800          2 INACTIVE
         3       5653   52428800          2 CURRENT
         2       5652   52428800          2 CURRENT
Here is the error from alertlog:
ORA-00600: internal error code, arguments: [3705], [1], [3], [2], [8], [], [], []

We had to perform point-in-time recovery up to 5652 sequence and DB felt much better :-)

Services, provided by my company:

Monthly Quarterly Annual
Consulting 8х5, includes: 49,99 € 119,99 € 319,99 €
Consulting + + +
Studying a test case (Except RAC) 2 5
Support 8х5, includes: 149,99 € 359,99 € 959,99 €
Consulting + + +
Studying a test case 3
(Except RAC)
6 9
Access to knowledge base + + +
Monitoring a DB instance (24х7) + +
Emergency troubleshooting +
Complete DB outsourcing (24х7)* 999,99 € 2 399,99 € ** 6 399,99 € **
Consulting + + +
Studying a test case + + +
Access to knowledge base + + +
Monitoring a DB instance + + +
Emergency troubleshooting + + +
DB migrating/patching + +

*    DB is limited up to 750MB in size and single instance, additionally an offline stand-by database can be maintaned. Ask for additional options.

** All payments can be proceeded via PayPal©, except Quarterly and Annual complete DB Outsourcing.

Fill free to contact us for info via e-mail or contact form.

Books: Oracle Data Guard 11gR2 Administration…

With all my pleasure I’m glad to announce my friend’s book  “Oracle Data Guard 11gR2 Administration Beginner’s Guide”.

All you need to know about disaster recovery technique you’ll find at these pages!

DG_Image Table of Contents

Preface
Chapter 1: Getting Started
Chapter 2: Configuring the Oracle Data Guard Physical Standby Database
Chapter 3: Configuring Oracle Data Guard Logical Standby Database
Chapter 4: Oracle Data Guard Broker
Chapter 5: Data Guard Protection Modes
Chapter 6: Data Guard Role Transitions
Chapter 7: Active Data Guard, Snapshot Standby, and Advanced Techniques
Chapter 8: Integrating Data Guard with the Complete Oracle Environment
Chapter 9: Data Guard Configuration Patching
Chapter 10: Common Data Guard Issues
Chapter 11: Data Guard Best Practices
Pop Quiz Answers
Index

  • Preface
  • Chapter 1: Getting Started
    • What is Data Guard?
    • Standby database
      • Physical standby database
      • Logical standby database
      • Snapshot standby database
  • Oracle Data Guard evolution
    • Version 7.3 – stone age
    • Version 8i – first age
    • Version 9i – middle age
    • Version 10g – new age
    • Version 11g – modern age
  • Oracle Data Guard architecture
    • Data Guard services
      • Redo transport services
      • Apply services
  • Time for action – monitoring Redo Apply
    • SQL Apply (logical standby databases)
  • Role transitions
    • Switchover
    • Failover
  • User interfaces for administering Data Guard
    • SQL*Plus
    • DGMGRL
    • Enterprise Manager
  • Time for action – using interfaces to monitor Data Guard
    • Data Guard background processes
  • Other replication solutions and Data Guard
    • Storage-based replication solutions
    • GoldenGate and Streams
  • Summary
  • Chapter 2: Configuring the Oracle Data Guard Physical Standby Database
    • Preconfiguration for Data Guard
      • Data loss consideration
      • Network bandwidth consideration
      • Preparing the primary database
        • Archive log mode
    • Time for action – enabling the archive log mode
      • Force logging
  • Time for action – enabling force logging
    • Standby redo logs
  • Time for action – configuring standby redo logs on primary
    • Fast recovery area (FRA)
  • Time for action – enabling FRA
    • Understanding initialization parameters
    • DB_NAME
    • DB_UNIQUE_NAME
    • LOG_ARCHIVE_CONFIG
    • LOG_ARCHIVE_MAX_PROCESSES
    • LOG_ARCHIVE_DEST_n
    • LOCATION and SERVICE
    • VALID_FOR
    • SYNC and ASYNC
    • AFFIRM and NOAFFIRM
    • COMPRESSION
    • MAX_CONNECTIONS
    • MAX_FAILURE
    • REOPEN
    • NET_TIMEOUT
    • DELAY
    • LOG_ARCHIVE_DEST_STATE_n
  • Creating the physical standby database
    • Standby database related initialization parameters
      • FAL_SERVER
      • STANDBY_FILE_MANAGEMENT
      • DB_FILE_NAME_CONVERT
      • LOG_FILE_NAME_CONVERT
    • The physical standby database instance
  • Time for action – starting the physical standby instance and making it ready for the RMAN duplicate
    • Using RMAN duplicate to create physical standby databases
  • Time for action – running an RMAN duplicate
  • Post-installation steps
    • Verifying the standby database configuration
  • Time for action – verifying the standby database configuration
    • Managing Redo Apply
  • Time for action – starting, stopping, and monitoring MRP
    • Verifying synchronization between the primary and standby databases
  • Time for action – verifying synchronization between the primary and standby databases
  • Time for action – testing real-time apply
  • Summary
  • Chapter 3: Configuring Oracle Data Guard Logical Standby Database
    • Logical standby database characteristics
      • Not everything must be duplicated
      • Use for reporting at all times
      • Independent standby database objects
      • Protecting writes on replicated standby tables
      • Limitation for specific data types and objects
      • High availability and disaster recovery considerations
    • Preparation for the configuration
    • Time for action – checking for the unsupported data types
    • Time for action – searching for and fixing any table row uniqueness problem
    • Creating a logical standby database
    • Time for action – making a physical standby database environment ready for conversion
    • Time for action – converting a physical standby database into a logical standby database
    • Verifying the logical standby database
    • Time for action – checking the redo transport service status
    • Time for action – checking the SQL Apply service status
    • Customization and management in a logical standby database
      • Selective replication in a logical standby database
    • Time for action – working with skip rules on a logical standby database
      • Data base Guard settings for the logical standby database
    • Time for action – changing the Database Guard setting
      • Disabling database guard for a session
    • Creating objects on the logical standby database
      • Creating and re-creating tables
      • Creating scheduler jobs
      • Creating materialized views
  • Time for action – creating objects on the logical standby database
    • Automatic deletion of archived logs
      • Deletion of the foreign archived logs
      • Deletion of the local archived logs
  • Summary
  • Chapter 4: Oracle Data Guard Broker
    • Introduction to Data Guard broker
    • Data Guard broker features and benefits
      • Centralized and simple management
      • Cloud Control integration
      • Oracle Data Guard and RAC
      • Role transition with Data Guard broker
      • Data Guard fast-start failover
        • Recommendation
    • Data Guard broker components
      • Oracle Data Guard broker server-side components
        • Data Guard Monitor process (DMON)
        • Configuration file
      • Oracle Data Guard broker client-side components
        • DGMGRL utility
        • Enterprise Manager Cloud Control client
    • Implementation of Oracle Data Guard broker
    • Time for action – initial setup of Data Guard broker
    • Time for action – connecting to Data Guard broker
    • Time for action – basic monitoring with Data Guard broker
    • Management with Data Guard broker
      • Enabling and disabling broker configuration
    • Time for action – disabling broker configuration
      • Enabling and disabling a standby database
    • Time for action – disabling and enabling database
      • Changing configuration and database properties using broker
    • Time for action – changing the database name
      • Changing the state of the database
      • Troubleshooting Data Guard broker
      • Data Guard tracing
    • Most Common Data Guard broker issues
      • ORA-16797: database is not using a server parameter file
      • ORA-10458:standby database requires recovery
      • ORA-16737:the redo transport service for standby database “string” has an error
      • ORA-16715:redo transport-related property string of standby database “string” is inconsistent
      • ORA-12514:TNS:listener does not currently know of service requested in connect descriptor
        • Current listener description
    • Oracle Data Guard fast-start failover
    • Time for action – configuring fast-start failover
      • Troubleshooting observer configuration
      • Script to stop and start observer
    • Summary
  • Chapter 5: Data Guard Protection Modes
    • The Maximum Protection mode
    • The Maximum Performance mode
    • The Maximum Availability mode
    • Choosing the correct mode for your requirements
    • Changing Data Guard protection mode
    • Time for action – changing the protection mode with SQL*Plus
    • Time for action – changing the protection mode with Data Guard broker
    • Time for action – changing the protection mode with Enterprise Manager Cloud Control
    • Summary
  • Chapter 6: Data Guard Role Transitions
    • Role transition considerations
    • Switchover
      • Performing switchover with a physical standby database using SQL*Plus
    • Time for action – preliminary tests before performing switchover
    • Time for action – switchover with a physical standby using SQL*Plus
      • Performing switchover with a physical standby database using broker
    • Time for action – switchover with a physical standby using broker
      • Performing switchover with a physical standby database using EM Cloud Control
    • Time for action – switchover with a physical standby using EM Cloud Control
      • Performing switchover with a logical standby database using SQL*Plus
    • Time for action – switchover with a logical standby database using SQL*Plus
      • Performing switchover with a logical standby database using broker
    • Time for action – switchover with a logical standby using broker
    • Failover
      • Performing failover with a physical standby database
    • Time for action – failover with a physical standby database using SQL*Plus
      • Performing failover with a logical standby database
    • Time for action – failover with a logical standby using broker
    • Summary
  • Chapter 7: Active Data Guard, Snapshot Standby, and Advanced Techniques
    • Oracle Active Data Guard
      • Why Active Data Guard?
      • Oracle Data Guard license
      • Enabling Active Data Guard
    • Time for action – enabling Active Data Guard if Redo Apply is running using SQL *PLUS
    • Time for action – enabling Active Data Guard if the standby database is shut down
    • Time for action – enabling Active Data Guard using broker
      • Monitoring Active Data Guard
        • From primary
        • From standby
      • Active Data Guard with applications
        • Active Data Guard with PeopleSoft
    • Time for action – Active Data Guard with PeopleSoft
      • Active Data Guard with EBS
      • Active Data Guard with TopLink
      • Active Data Guard with Oracle BI
      • Active Data Guard with SAP
    • Active Data Guard features
      • EXPDP from standby database using NETWORK_LINK (ADG)
  • Time for action – exporting a database backup from Active Data Guard
  • Time for action – using the ASH report from the standby database
  • Using a snapshot standby database
  • Time for action – converting to a snapshot standby database
  • Time for action – converting to a physical standby database
  • Cascade standby databases
    • Limitations with cascade standby database
  • Time for action – cascade standby database
  • Advanced compression in Data Guard
  • Time for action – enabling advanced compression
  • Preparation of standby on a cross-platform Data Guard
  • Time for action – creating a cross-platform Data Guard setup
  • Data Guard tuning and wait events
    • Network tuning
    • Redo transport and apply tuning
    • Data Guard wait events
  • Summary
  • Chapter 8: Integrating Data Guard with the Complete Oracle Environment
    • The Oracle Enterprise Manager Cloud Control integration
    • Time for action – adding the Data Guard configuration into Cloud Control
      • Cloud Control Data Guard administration home page
      • Modifying the Data Guard configuration
    • Time for action – enabling/disabling fast-start failover
      • Monitoring Data Guard performance
      • Using Incident Manager to monitor Data Guard
    • Time for action – setting the threshold and creating an incident for estimated failover time metric
    • RMAN integration
      • Integration requirements and best practices
        • Physical standby requirement
        • RMAN Catalog requirement
        • Using a different DB_UNIQUE_NAME
        • General RMAN best practices
      • RMAN settings for the Data Guard environment
        • Registering primary database in the catalog
        • Configuring RMAN settings for primary database:
        • Configuring RMAN settings for standby database
        • Checking the RMAN configuration
    • Time for action – recovering a primary database using a standby database disk backup
      • Using block change tracking with Data Guard
    • RAC integration
      • A RAC primary database with a single instance standby database
      • A RAC primary database with a RAC standby database
    • Summary
  • Chapter 9: Data Guard Configuration Patching
    • What is patch and what are patch types?
      • Interim patch
      • CPU/SPU patches
      • PSU patches
      • Patch set
      • Patching on Data Guard
    • Best practices of patching
    • Upgrading OPatch
    • Performing prerequisite checks of patch
    • How to clean up patch history?
    • Patching on Data Guard configuration
      • How to apply an interim/bug patch on logical standby?
    • Time for action – applying a patch on logical standby
      • How to apply a PSU patch on physical standby database using broker?
    • Time for action – applying PSU on a physical standby database
      • How to apply patch set on physical standby (11.2.0.1 to 11.2.0.3)?
    • Time for action – patch set upgrade of physical standby
    • Summary
  • Chapter 10: Common Data Guard Issues
    • Recreating the standby control file
    • Time for action – recreating the standby control file
    • Dealing with redo transport authentication problems
    • Time for action – changing the SYS password in a Data Guard environment
    • Time for action – changing the redo transport user
    • Dealing with UNNAMED datafiles
    • Time for action – resolving UNNAMED datafile errors
    • Closing a gap with an RMAN incremental backup
    • Time for action – closing a gap with an RMAN incremental backup
    • Fixing NOLOGGING changes on the standby database
    • Time for action – fixing NOLOGGING changes on a standby database with incremental datafile backups
    • Time for action – fixing NOLOGGING changes in the standby database with incremental database backups
    • Turning on Data Guard tracing
    • Gathering diagnostic data
      • Alert log and trace files
    • Time for action – monitoring the database alert log using ADRCI
      • Data Guard broker logs
      • Dynamic performance views
    • Summary
  • Chapter 11: Data Guard Best Practices
    • Configuring a connection failover
      • Transparent Application Failover (TAF)
        • Configuring the client-side TAF
        • Configuring the server-side TAF
      • Fast Connection Failover (FCF)
    • Time for action – configuring FCF for JDBC connections
      • Fast Application Notification (FAN)
    • The archived log deletion policy on the standby database
    • Time for action – the recommended configuration for archived log maintenance on a standby database
    • Using flashback on a standby database
    • Time for action – using flashback on a standby database
    • Database rolling upgrade using the transient logical standby database
    • Time for action – performing a rolling upgrade using the transient logical standby database
    • Corruption detection, prevention, and automatic repair with Oracle Data Guard
      • DB_BLOCK_CHECKSUM
      • DB_BLOCK_CHECKING
      • DB_LOST_WRITE_PROTECT
      • Automatic block media repair
    • Summary
  • Pop Quiz Answers
    • Chapter 1, Getting Started
    • Chapter 5, Data Guard Protection Modes
    • Chapter 9, Data Guard Configuration Patching
    • Chapter 10, Common Data Guard Issues
  • Index

Useful scripts: Get DDL for jobs, created by DBMS_JOB…

Here is a script, which is dedicated to generate DDL commands to recreate jobs, created by DBMS_JOB package.

DECLARE
 CURSOR job_list
 IS
 SELECT JOB FROM dba_jobs;
mysql VARCHAR2 (32767);
BEGIN
 FOR job_id IN job_list
 LOOP
 mysql := '';
 DBMS_OUTPUT.put_line ('BEGIN');
 DBMS_OUTPUT.put_line ('DBMS_JOB.REMOVE(' || job_id.job || ');');
 DBMS_JOB.user_export (job_id.job, mysql);
 DBMS_OUTPUT.put_line (mysql);
 DBMS_OUTPUT.put_line ('END;');
 DBMS_OUTPUT.put_line ('/');
 END LOOP;
END;
/

Useful scripts: Log file switch frequency statistsic

Very nice and easy to use query.

SELECT trunc(first_time) "Date",
 to_char(first_time, 'Dy') "Day",
 count(1) "Total",
 SUM(decode(to_char(first_time, 'hh24'), '00', 1, 0)) "h0",
 SUM(decode(to_char(first_time, 'hh24'), '01', 1, 0)) "h1",
 SUM(decode(to_char(first_time, 'hh24'), '02', 1, 0)) "h2",
 SUM(decode(to_char(first_time, 'hh24'), '03', 1, 0)) "h3",
 SUM(decode(to_char(first_time, 'hh24'), '04', 1, 0)) "h4",
 SUM(decode(to_char(first_time, 'hh24'), '05', 1, 0)) "h5",
 SUM(decode(to_char(first_time, 'hh24'), '06', 1, 0)) "h6",
 SUM(decode(to_char(first_time, 'hh24'), '07', 1, 0)) "h7",
 SUM(decode(to_char(first_time, 'hh24'), '08', 1, 0)) "h8",
 SUM(decode(to_char(first_time, 'hh24'), '09', 1, 0)) "h9",
 SUM(decode(to_char(first_time, 'hh24'), '10', 1, 0)) "h10",
 SUM(decode(to_char(first_time, 'hh24'), '11', 1, 0)) "h11",
 SUM(decode(to_char(first_time, 'hh24'), '12', 1, 0)) "h12",
 SUM(decode(to_char(first_time, 'hh24'), '13', 1, 0)) "h13",
 SUM(decode(to_char(first_time, 'hh24'), '14', 1, 0)) "h14",
 SUM(decode(to_char(first_time, 'hh24'), '15', 1, 0)) "h15",
 SUM(decode(to_char(first_time, 'hh24'), '16', 1, 0)) "h16",
 SUM(decode(to_char(first_time, 'hh24'), '17', 1, 0)) "h17",
 SUM(decode(to_char(first_time, 'hh24'), '18', 1, 0)) "h18",
 SUM(decode(to_char(first_time, 'hh24'), '19', 1, 0)) "h19",
 SUM(decode(to_char(first_time, 'hh24'), '20', 1, 0)) "h20",
 SUM(decode(to_char(first_time, 'hh24'), '21', 1, 0)) "h21",
 SUM(decode(to_char(first_time, 'hh24'), '22', 1, 0)) "h22",
 SUM(decode(to_char(first_time, 'hh24'), '23', 1, 0)) "h23",
 round(count(1) / 24, 2) "Avg"
 FROM V$log_history
 group by trunc(first_time), to_char(first_time, 'Dy')
 Order by 1

Useful scripts: Catch a session, causing ‘cursor: pin S wait on X’…

For sure, every network from time to time aborts connection. Especially this commonly could be seen in WANs.

When this happens while a session is fetching data or using some other object through db-link, PMON for some reason (from time to time) is unable to kill this session, so session stucks, leading to ‘cursor: pin S wait on X’ wait event.

It’s quite difficult to find out, which session you need to kill to release memory lock, so here is the query for this:

SELECT *
 FROM v$session
 WHERE sid IN
 (SELECT TO_NUMBER (SUBSTR (TO_CHAR (RAWTOHEX (p2raw)), 1, 8),
 'XXXXXXXX') sid
 FROM v$session
 WHERE event = 'cursor: pin S wait on X')

The following screenshot shows reducing Latch/mutex waits:

Help, my DB is feeling ill: Troubleshooting sequence…

Hello, my friends!

Many times I’d been asked for a reasons, due to which DB could stuck or even be aborted, so I decided to make a little overview. When a DB hangs, stops responding or even doesn’t startup, then a typical behaviour for a novice – running in circle and screaming.

First thing you must to do – just relax, all bad things, that could happen, already have happen. Take a breath, make a cup of coffee and relax: it’s time for diagnostics. We must find out a problem and allow users to work as fast as possible, all analysis  we’ll make afterwards.

My troubleshooting sequence is:

    1. Hardware;
    2. OS;
    3. Instance;
    4. Database;
    5. Network;
    6. Oracle.Net;
  1. Hardware. In most cases this work is for system administrators: if server is down, DBA can’t even connect to shell.  But nevertheless from time to time this part is assigned to DBA:
    1. Check for error messages at hardware monitoring screens, if there is any;
    2. Check OS logs.
  2. OS. The same – this also belongs to SA. But OS environment parameters must be set by DBA:
    1. Check if Oracle parameters ($ORACLE_HOME,$ORACLE_SID,NLS_LANG etc) are set correctly;
    2. Take a look at OS logs for any related errors;
    3. Check, if OS has enough resources for running and maintaining Oracle DB. Poor resources can make new connections impossible.
  3. Instance:
    1. Find out, if it is started and in which mode;
    2. Take a look at alert log and search for ORA errors that prevent instance to start up or open;
    3. Performance issue, that can stuck all activity (this topic I’ll describe in my next article);
    4. Don’t forget about resources: Memory parameters, Flash Recovery Area free space (a common error of young DBA).
  4. Database. If instance can’t open DB, then there could be a quite badly situation with DB files:
    1. Check alert log for ORA errors;
    2. Check for control file loss;
    3. Check for DB files corruptions or loss of log file groups.
  5. Network:
    1. Check, if NIC is up and configured;
    2. Check, if server can reach its default router and vice versa;
    3. Check, if server can be reached by client PC.
  6. Oracle.Net:
    1. Try to tnsping server and analyze an error if any;
    2. Check, if Listener is up and configured;
    3. Check for a list of registered databases;
    4. Check, if naming method at client’s PC is configured correctly;
    5. Take a look at Oracle.Net log files (listener.ora, sqlnet.log).

As you can see, the most frequently mentioned file is alert log. So if server is up and OS is running, alert log is the first place to look at.

Useful scripts: Get a list of objects to be moved for a data file resize

Every DBA from time to time wants to reduce a size of a data file, thinking ‘There is a lot of free space in datafile, why don’t to resize it?’ But the secret is that shrinking tables is not very helpful in this case – datafile least size will be limited to the block, where the last table’s/index’s block resides.

So, the solution is to relocate used blocks to free gaps between blocks.

We can achieve this by several ways:

  • ALTER TABLE …. MOVE or INDEX REDUILD;
  • use DBMS_REDIFINITION package;
  • Export table, purge it and import again with the help of DataPump.

But prior we need to get a list of objects, which need to be relocated to make datafile reduce possible. The following query show such objects:

DECLARE
   V_FILE_ID       NUMBER;
   V_BLOCK_SIZE    NUMBER;
   V_RESIZE_SIZE   NUMBER;
BEGIN
   V_FILE_ID := &FILE_ID;
   V_RESIZE_SIZE := &RESIZE_FILE_TO;
   SELECT BLOCK_SIZE
     INTO V_BLOCK_SIZE
     FROM V$DATAFILE
    WHERE FILE# = V_FILE_ID;

   DBMS_OUTPUT.PUT_LINE ('.');
   DBMS_OUTPUT.PUT_LINE ('.');
   DBMS_OUTPUT.PUT_LINE ('.');
   DBMS_OUTPUT.PUT_LINE (
         'OBJECTS IN FILE '
      || V_FILE_ID
      || ' THAT MUST MOVE IN ORDER TO RESIZE THE FILE TO '
      || V_RESIZE_SIZE
      || ' BYTES');
   DBMS_OUTPUT.PUT_LINE (
      '=======================================================================');
   FOR my_record
      IN (  SELECT DISTINCT
                   (   OWNER
                    || '.'
                    || SEGMENT_NAME
                    || ' - OBJECT TYPE = '
                    || SEGMENT_TYPE)
                      ONAME
              FROM DBA_EXTENTS
             WHERE (block_id + blocks - 1) * V_BLOCK_SIZE > V_RESIZE_SIZE
                   AND FILE_ID = V_FILE_ID
          ORDER BY 1)
   LOOP
      DBMS_OUTPUT.PUT_LINE (my_record.ONAME);
   END LOOP;
END;
/

Useful scripts: Table list with stale statistics

This script is intended for daily use to get tables where percentage of changed records is above 10%:

SELECT DT.OWNER,
       DT.TABLE_NAME,
       ROUND ( (DELETES + UPDATES + INSERTS) / NUM_ROWS * 100) PERCENTAGE
FROM   dba_tables dt, ALL_TAB_MODIFICATIONS atm
WHERE      DT.OWNER = ATM.TABLE_OWNER
       AND DT.TABLE_NAME = ATM.TABLE_NAME
       AND NUM_ROWS > 0
       AND ROUND ( (DELETES + UPDATES + INSERTS) / NUM_ROWS * 100) >= 10
ORDER BY 3 desc

Also you can avoid system schemas by adding:

AND OWNER NOT IN ('SYS','SYSTEM','DBSNMP','OSMMON','PERFSTAT')
Follow

Get every new post delivered to your Inbox.

Join 42 other followers

%d bloggers like this: