Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-05-15
15:39:51 openstackgerrit Stephen Finucane proposed openstack/nova master: tox: Integrate mypy https://review.opendev.org/676208
15:39:52 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Add typing information https://review.opendev.org/714695
15:39:52 openstackgerrit Stephen Finucane proposed openstack/nova master: hardware: Update and correct typing information https://review.opendev.org/714694
15:39:53 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Replace 'cpu_pinning_requested' helper https://review.opendev.org/714697
15:39:53 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Split instance NUMA object tests https://review.opendev.org/714696
15:39:54 openstackgerrit Stephen Finucane proposed openstack/nova master: hardware: Remove handling of pre-Train compute nodes https://review.opendev.org/714699
15:39:54 openstackgerrit Stephen Finucane proposed openstack/nova master: hardware: Don't consider overhead CPUs for unpinned instances https://review.opendev.org/714698
15:39:55 openstackgerrit Stephen Finucane proposed openstack/nova master: hardware: Tweak the 'cpu_realtime_mask' handling slightly https://review.opendev.org/461456
15:39:55 openstackgerrit Stephen Finucane proposed openstack/nova master: hardware: Add validation for 'cpu_realtime_mask' https://review.opendev.org/468203
15:39:56 openstackgerrit Stephen Finucane proposed openstack/nova master: hardware: Invert order of NUMA topology generation https://review.opendev.org/714701
15:39:56 openstackgerrit Stephen Finucane proposed openstack/nova master: hardware: Rework 'get_realtime_constraint' https://review.opendev.org/714700
15:39:57 openstackgerrit Stephen Finucane proposed openstack/nova master: hardware: Remove '_numa_fit_instance_cell_with_pinning' https://review.opendev.org/714703
15:57:04 openstackgerrit Huaqiang Wang proposed openstack/nova master: Introduce 'pcpuset' field for InstanceNUMACell https://review.opendev.org/714658
15:57:05 openstackgerrit Huaqiang Wang proposed openstack/nova master: Calculate the CPU usage for mixed instance https://review.opendev.org/716269
15:57:05 openstackgerrit Huaqiang Wang proposed openstack/nova master: Introduce the 'CPUAllocationPolicy.MIXED' enum https://review.opendev.org/716267
15:57:06 openstackgerrit Huaqiang Wang proposed openstack/nova master: libvirt/hardware: refactor numa_get_constraints https://review.opendev.org/718294
15:57:06 openstackgerrit Huaqiang Wang proposed openstack/nova master: online data migration for cpuset of dedicated instance https://review.opendev.org/718493
15:57:07 openstackgerrit Huaqiang Wang proposed openstack/nova master: metadata: export the vCPU IDs that are pinning on the host CPUs https://review.opendev.org/688936
15:57:07 openstackgerrit Huaqiang Wang proposed openstack/nova master: Setup 'mixed' instance through 'PCPU' and 'VCPU' resource https://review.opendev.org/713355
15:57:08 openstackgerrit Huaqiang Wang proposed openstack/nova master: libvirt: apply mixed instance CPU policy https://review.opendev.org/728481
15:57:08 openstackgerrit Huaqiang Wang proposed openstack/nova master: notification: add the pcpuset info to instance numa cell payload https://review.opendev.org/728480
15:57:09 openstackgerrit Huaqiang Wang proposed openstack/nova master: flavor: Enable 'hw:cpu_dedicated_mask' for creating a mixed instance https://review.opendev.org/728483
15:57:09 openstackgerrit Huaqiang Wang proposed openstack/nova master: scheduler: mixed instance CPU resource translating https://review.opendev.org/728482
15:57:10 openstackgerrit Huaqiang Wang proposed openstack/nova master: hardware: create 'mixed' instance for realtime and shared CPUs https://review.opendev.org/728485
15:57:10 openstackgerrit Huaqiang Wang proposed openstack/nova master: compute: bump nova-compute service version for mixed instance https://review.opendev.org/728484
16:05:20 openstackgerrit Stephen Finucane proposed openstack/nova master: Add '__version__' attribute https://review.opendev.org/728490
16:14:33 openstackgerrit Andreas Jaeger proposed openstack/nova master: Switch to newer openstackdocstheme and reno versions https://review.opendev.org/728263
16:17:49 openstackgerrit Balazs Gibizer proposed openstack/nova master: Reserve DISK_GB resource for the image cache https://review.opendev.org/728439
16:18:25 gibi aarents, dansmith: this time with tests ^^
16:19:25 gibi and that marks the end of my work week. Have a nice weekend o/
16:40:16 openstackgerrit Huaqiang Wang proposed openstack/nova master: notification: add the pcpuset info to instance numa cell payload https://review.opendev.org/728480
16:40:16 openstackgerrit Huaqiang Wang proposed openstack/nova master: Introduce 'pcpuset' field for InstanceNUMACell https://review.opendev.org/714658
16:40:17 openstackgerrit Huaqiang Wang proposed openstack/nova master: libvirt: apply mixed instance CPU policy https://review.opendev.org/728481
16:40:17 openstackgerrit Huaqiang Wang proposed openstack/nova master: Introduce the 'CPUAllocationPolicy.MIXED' enum https://review.opendev.org/716267
16:40:18 openstackgerrit Huaqiang Wang proposed openstack/nova master: Calculate the CPU usage for mixed instance https://review.opendev.org/716269
16:40:18 openstackgerrit Huaqiang Wang proposed openstack/nova master: computer: validate cpu pinning configuration for mixed instance https://review.opendev.org/716268
16:40:19 openstackgerrit Huaqiang Wang proposed openstack/nova master: scheduler: mixed instance CPU resource translating https://review.opendev.org/728482
16:40:19 openstackgerrit Huaqiang Wang proposed openstack/nova master: online data migration for cpuset of dedicated instance https://review.opendev.org/718493
16:40:20 openstackgerrit Huaqiang Wang proposed openstack/nova master: compute: bump nova-compute service version for mixed instance https://review.opendev.org/716271
16:40:20 openstackgerrit Huaqiang Wang proposed openstack/nova master: flavor: Enable 'hw:cpu_dedicated_mask' for creating a mixed instance https://review.opendev.org/714706
16:40:21 openstackgerrit Huaqiang Wang proposed openstack/nova master: libvirt/hardware: refactor numa_get_constraints https://review.opendev.org/718294
16:40:21 openstackgerrit Huaqiang Wang proposed openstack/nova master: hardware: create 'mixed' instance for realtime and shared CPUs https://review.opendev.org/728485
16:40:22 openstackgerrit Huaqiang Wang proposed openstack/nova master: metadata: export the vCPU IDs that are pinning on the host CPUs https://review.opendev.org/688936
16:40:22 openstackgerrit Huaqiang Wang proposed openstack/nova master: Setup 'mixed' instance through 'PCPU' and 'VCPU' resource https://review.opendev.org/713355
16:49:12 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934
16:49:12 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: Add emulated TPM support to Nova https://review.opendev.org/631363
16:56:00 openstackgerrit Stephen Finucane proposed openstack/nova-specs master: Encrypted Emulated Virtual TPM https://review.opendev.org/728505
18:42:29 mnaser hmm
18:42:45 mnaser i have a query that's absolutely destroying my db server and it seems pretty silly looking at it
18:42:56 mnaser https://www.irccloud.com/pastebin/RZ58u5w0/
18:43:19 mnaser i literally have 2 cells inside cell_mappings, yet im having to scan all of the instance mappings
18:44:00 mnaser explain in my case shows scanning 1.2m rows. yes, we could use an archive, but..
18:44:06 mnaser https://www.irccloud.com/pastebin/vmPO2rNN/
18:45:23 mnaser it sounds like this is the root: https://github.com/openstack/nova/blob/master/nova/objects/cell_mapping.py#L275-L285
18:46:16 mnaser which i guess comes from https://github.com/openstack/nova/blob/master/nova/compute/instance_list.py#L150-L153
18:52:40 mnaser which leads to more interesting things. instance_mappings seem to be not soft deleted yet i have 2586748 records...
19:23:32 dansmith mnaser: instance mappings are not soft deleted,
19:23:38 dansmith and are only ever removed when you archive
19:24:29 mnaser dansmith: ah. i'll try and add archive to happen more often because i think teh annoying thing is once a cloud grows big enough, the archive really hurt in terms of performance
19:25:09 dansmith yeah, it's not a very lightweight thing
19:25:40 dansmith mnaser: I guess we need to filter that instance mappings query by deleted eh?
19:25:51 dansmith er, no we can't
19:25:52 dansmith duh
19:25:54 mnaser yeah :(
19:26:03 mnaser i mean there's the queued_for_delete, i saw a lot of records with that
19:26:13 dansmith you'd think it would be pretty fast though
19:26:27 dansmith becase it's just distinct cell_id, and then filtered by project
19:26:34 dansmith maybe we're missing an index
19:27:08 mnaser the select distinct is def what hurts, im trying to run it manually and see how long it takes..
19:27:21 mnaser 16.02 seconds
19:27:29 dansmith there is an index on project_id, but not on cell_id
19:27:42 dansmith so maybe that is making the distinct take a long time?
19:27:55 mnaser dansmith: i think the other problem is that t his code is also used for quota checks, so thats ~16s per *single* request to boot an instance
19:28:16 mnaser when we'd see a big surge of instance creations, that's 16s*20 or so and it starts getting worse
19:28:34 mnaser specifically here https://github.com/openstack/nova/blob/575a91ff5be79ac35aef4b61d84c78c693693304/nova/quota.py#L1194-L1195
19:28:55 dansmith apparently mysql always indexes foreign key columns, so cell_id should be indexed
19:29:08 mnaser https://www.irccloud.com/pastebin/MftvYe3g/
19:29:10 mnaser thats the EXPLAIN
19:29:23 dansmith mnaser: you specifically have the subset conf turned on, is that intentional?
19:29:53 mnaser dansmith: sorry, nto following, subset conf?
19:30:09 dansmith mnaser: see the comment just above the code in instance_list ?
19:30:16 dansmith query_cell_subset = CONF.api.instance_list_per_project_cells
19:30:25 dansmith if that is off then it won't do this
19:31:01 dansmith defaults to false
19:31:36 dansmith heh, the conf text (written by me I'm sure) says turning that on "causes one more (fast) query to the API database"
19:31:48 mnaser heh, let me check if its enabled
19:31:55 dansmith maybe I should edit that comment or say "assuming you're archiving regularly" :)
19:32:08 mnaser instance_list_per_project_cells is not set to anything so its probably set to default False
19:32:26 dansmith mnaser: okay well you see the code, not sure how you could be runningthat
19:32:41 mnaser dansmith: https://github.com/openstack/nova/blob/575a91ff5be79ac35aef4b61d84c78c693693304/nova/quota.py#L1194-L1195 ?
19:32:55 mnaser that doesnt seem to respect it
19:33:04 dansmith mnaser: no, the instance list one
19:33:17 dansmith maybe you're not actually hitting the instance_list code, just the quota one?
19:33:29 mnaser dansmith: yes, sorry, should have clarified, i think that's the running theory
19:33:36 dansmith okay I thought you assumed both
19:33:44 mnaser i'm looking at logs and os_volume-boot is taking some ~45-60s-ish to respond
19:33:44 dansmith I wrote instance_list so was focusing there
19:33:56 mnaser yeah, initially that was my two findings, but i agree, it seems quota code might be what is hurting
19:34:03 dansmith gotcha
19:34:05 dansmith melwitt: ^

Earlier   Later