Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-07-14
14:56:32 dansmith gmann: tres woot: https://review.opendev.org/#/c/734184/
14:57:58 gmann dansmith: nice, checking
14:58:26 dansmith (moved to -qa)
16:09:50 openstackgerrit Balazs Gibizer proposed openstack/nova master: [WIP] Support SRIOV interface attach and detach https://review.opendev.org/740995
16:10:44 gibi stephenfin, sean-k-mooney: this might be intresting for you ^^
16:12:10 openstackgerrit Dan Smith proposed openstack/nova master: Make our ceph job test with glance in multistore mode https://review.opendev.org/734184
16:42:31 openstackgerrit Huaqiang Wang proposed openstack/nova master: objects: Introduce the 'CPUAllocationPolicy.MIXED' enum https://review.opendev.org/716267
16:42:31 openstackgerrit Huaqiang Wang proposed openstack/nova master: notifications: add the pcpuset info to instance numa cell payload https://review.opendev.org/728480
16:42:32 openstackgerrit Huaqiang Wang proposed openstack/nova master: libvirt: apply mixed instance CPU policy https://review.opendev.org/728481
16:42:32 openstackgerrit Huaqiang Wang proposed openstack/nova master: hardware: refactor the code for CPU policy sanity check https://review.opendev.org/740925
17:08:01 openstackgerrit Huaqiang Wang proposed openstack/nova master: Calculate the CPU usage for mixed instance https://review.opendev.org/716269
17:08:02 openstackgerrit Huaqiang Wang proposed openstack/nova master: scheduler: mixed instance CPU resource translating https://review.opendev.org/728482
17:25:55 openstackgerrit Huaqiang Wang proposed openstack/nova master: Validate CPU pinning configuration for mixed instance https://review.opendev.org/716268
17:25:56 openstackgerrit Huaqiang Wang proposed openstack/nova master: scheduler: mixed instance CPU resource translating https://review.opendev.org/728482
17:25:56 openstackgerrit Huaqiang Wang proposed openstack/nova master: Calculate the CPU usage for mixed instance https://review.opendev.org/716269
18:53:20 openstackgerrit Merged openstack/nova master: utils: Move 'get_bdm_image_metadata' to nova.block_device https://review.opendev.org/739996
20:30:54 dansmith sean-k-mooney: replied. thanks for reviewing and for your initial help in that of course
20:33:00 sean-k-mooney no worries
20:36:34 sean-k-mooney dansmith: ya makes sense to me
20:36:55 sean-k-mooney im glad you finally got it working by the way.
20:37:06 sean-k-mooney its always nice when the ci verifys something like this for us
20:37:47 sean-k-mooney the glance folks might even want to take a look for there own testing
21:36:25 openstack bug 1550919 in OpenStack Compute (nova) rocky "[Libvirt]Evacuate fail may cause disk image be deleted" [Medium,In progress] https://launchpad.net/bugs/1550919 - Assigned to Lee Yarwood (lyarwood)
21:36:25 openstackgerrit Merged openstack/nova stable/rocky: Add functional test for bug 1550919 https://review.opendev.org/732707
21:54:44 dansmith sean-k-mooney: yeah, they'll have a lot more surface they can test once my two devstack patches land.. I hope they will make use of that, but.. who knows
22:24:03 sean-k-mooney dansmith: well if we are testing it its better then nothing
23:20:46 openstackgerrit Merged openstack/nova stable/rocky: libvirt: Don't delete disks on shared storage during evacuate https://review.opendev.org/732708
23:37:53 mnaser melwitt: i know you've worked some with the quota code. does nova have any locking on quota usages? neutron currently has a row-level lock on a resource/tenant_id which is causing me worlds of pain on highly parallelized port creation
23:38:24 mnaser it pretty much holds a lock the whole time (with a select for update) while it calculates the usage and updates...
23:46:10 gmann dansmith: seems oslo policy not able to format the target data for copy_image - https://zuul.opendev.org/t/openstack/build/aa1a5cd62aaa4210b19a5877e77b87db/log/controller/logs/screen-g-api.txt#600
23:47:21 gmann trying to reproduce this locally to see if oslo trying to change 'hw_rng_model' for some reason.
23:52:06 gmann mnaser: not sure about lock (melwitt can confirm if we do have that) but nova does recheck quota again just before the resource creation to avoid(or to minimize as much as possible) any wrong calculation or over-quota in parallel request cases.
23:52:45 mnaser gmann: ok so nova doesnt do any row level locking for quota usage -- also yeah, i remember that, it's even an option to recheck quota in config if i remember right
23:53:28 gmann * after resource creation. this config - https://github.com/openstack/nova/blob/7385adb2f526166bb174dab055d060a428d4d4ff/nova/conf/quota.py#L189
23:56:04 gmann mnaser: afaik, no locks but melwitt or dansmith can confirm it.
#openstack-nova - 2020-07-15
01:09:23 melwitt mnaser: I implemented a re-architecting of quotas in pike, which began counting quota usage on-the-fly by counting resources (this only does a read). prior to pike, we did do the select-for-update locking thing
02:26:53 openstackgerrit Jeffrey Zhang proposed openstack/nova master: Support inject-nmi action in watchdog https://review.opendev.org/741072
03:18:04 openstackgerrit Merged openstack/nova master: doc: Correct typo https://review.opendev.org/740670
03:32:41 openstackgerrit Keigo Noha proposed openstack/nova master: Change default num_retries for glance to 3 https://review.opendev.org/740389
05:38:27 openstackgerrit Jeffrey Zhang proposed openstack/nova master: Support inject-nmi action in watchdog https://review.opendev.org/741072
06:22:49 openstackgerrit Jie Li proposed openstack/nova-specs master: add volume backed server rebuild spec https://review.opendev.org/739349
08:50:12 openstackgerrit Huaqiang Wang proposed openstack/nova master: hardware: Enable 'hw:cpu_dedicated_mask' for creating a mixed instance https://review.opendev.org/714706
10:13:49 dpaclt Hi All I am unable to launch new vms .. http://paste.openstack.org/show/795937/ ..Anyone please suggest
10:15:50 lyarwood dpaclt: this isn't a support channel but grep for the req-4ce3ba68-34f8-453c-bb2a-9be9ef9adcbd id across the api/scheduler/conductor logs and you should see why all of your hosts are unable to schedule the instance
10:18:01 lyarwood elod: https://review.opendev.org/#/c/732717/ (and the two under it) is finally ready btw if you have time today
10:19:00 dpaclt Thanks lyarwood
10:44:39 elod lyarwood: sure, added to my TODOs :)
10:45:25 lyarwood thanks!
10:49:07 openstackgerrit Merged openstack/nova master: objects: Introduce 'pcpuset' field for InstanceNUMACell https://review.opendev.org/714658
12:01:16 openstack bug 1550919 in OpenStack Compute (nova) queens "[Libvirt]Evacuate fail may cause disk image be deleted" [Medium,In progress] https://launchpad.net/bugs/1550919 - Assigned to Lee Yarwood (lyarwood)
12:01:16 openstackgerrit Elod Illes proposed openstack/nova stable/queens: Add functional test for bug 1550919 https://review.opendev.org/732715
12:01:51 openstackgerrit Elod Illes proposed openstack/nova stable/queens: libvirt: Fix misleading debug msg "Instance is running" https://review.opendev.org/732716
12:10:16 openstackgerrit Elod Illes proposed openstack/nova stable/queens: libvirt: Don't delete disks on shared storage during evacuate https://review.opendev.org/732717
12:16:55 jsuchome gibi: hi, could you take another look at https://review.opendev.org/#/c/738738/ ?
12:27:08 devfaz Hi, is this a bug or known. If I try to disable an nova-compute service of a system terminated just seconds ago I get a trace with "An error occurred while updating the COMPUTE_STATUS_DISABLED trait on compute node resource providers managed by host XX. The trait will be synchronized automatically by the compute service when the update_available_resource periodic task runs" and rabbitmq-msgid timeout.
12:27:27 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.PY2 and six.PY3 https://review.opendev.org/726512
12:27:41 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.add_metaclass https://review.opendev.org/726565
12:28:01 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.reraise https://review.opendev.org/726898
12:28:20 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.moves https://review.opendev.org/727224
12:29:00 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.iteritems/itervalues/iterkeys https://review.opendev.org/727757
12:29:19 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.byte2int/int2byte https://review.opendev.org/727777
12:29:41 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.binary_type/integer_types/string_types https://review.opendev.org/728094
12:40:28 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.text_type (1/2) https://review.opendev.org/728109
13:00:08 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.text_type (2/2) https://review.opendev.org/728117
13:02:23 openstack Launchpad bug 1858667 in OpenStack Compute (nova) "TestMultiCellMigrate.test_poll_unconfirmed_resizes_with_(no_)upcall race failing with greenlet error: Error auto-confirming resize: Reader <greenlet.greenlet object at 0x7f55c19dd048> to writer privilege escalation not allowed. Will retry later." [Undecided,Confirmed]
13:02:23 lyarwood dansmith: https://bugs.launchpad.net/nova/+bug/1858667 - I'm hitting this odd locking issue in some func tests, this smells like some behaviour has changed in py38 tbh but I'm having a hard time pinning it down. Would you be able to take a look?
13:21:02 lyarwood melwitt: ^ also if you have any time, I'm not even sure why we are wrapping target_cell if I'm honest
13:21:35 lyarwood CELL_CACHE is the only global thing I can see?
13:30:55 dansmith lyarwood: the entire DB state is global, which is what the lock is protecting
13:31:29 dansmith lyarwood: your python REPL examples aren't representative I think because the lock grabbing is serial in the code, yet you're nesting them (I think, without indentation)
13:32:20 dansmith I'm not sure what the references to fasteners has to do with this, as we're using regular python.threading locks, IIRC
13:35:05 dansmith if I had to guess, I'd say maybe we're recursing on the lock somewhere, i.e. targeting the cell and then calling something that targets the cell again?
13:35:35 dansmith ah lockutils uses fasteners, I see
13:35:43 lyarwood dansmith: yeah sorry back
13:35:58 lyarwood dansmith: and yeah I did use indentation, launchpad-- just removed it for some reason
13:36:20 lyarwood dansmith: but yes my examples are bogus given what is actually being called
13:36:33 dansmith ack
13:36:49 lyarwood dansmith: I just hit the error and got carried away :)
13:37:40 dansmith is that in a periodic? poll unconfirmed resizes?
13:38:41 dansmith I may be making this up, but I think melwitt might have added something much later than this to make the periodics run targeted to the right cell, so maybe that's where the conflict is here... this is now getting lucky and running a periodic when we previously didn't need to change the target or something
13:38:44 lyarwood yeah
13:48:32 masterpe I try to live migrate a instance, but I get the message in the controller: "Unable to find record for source node compute40.domain.tld on compute48"
13:48:45 masterpe but compute40 is incorrect
13:49:03 masterpe What is wrong?
13:56:39 sean-k-mooney that looks like maybe the instace.host is wrong
13:56:59 sean-k-mooney which can happen if you have previous failed migrations in post live migrate
13:57:12 sean-k-mooney but im not sure
13:58:09 sean-k-mooney masterpe: i would check that the vm is actully on the host nova thinks its on
14:05:38 masterpe If I do a openstack server show instanceID:
14:05:41 masterpe OS-EXT-SRV-ATTR:host | compute48
14:05:48 masterpe OS-EXT-SRV-ATTR:hypervisor_hostname | compute40.domain.tld
14:08:35 openstackgerrit Huaqiang Wang proposed openstack/nova master: compute: bump nova-compute version and check in API https://review.opendev.org/740933
14:08:36 openstackgerrit Huaqiang Wang proposed openstack/nova master: hardware: create 'mixed' instance for realtime and shared CPUs https://review.opendev.org/728485
14:10:52 masterpe sean-k-mooney: How to correct this? I have try to do a "update instances set node = 'compute48.domain.tld' where uuid = 'e848a42b-a7b2-4885-a290-34da7beb98ad';"
14:11:05 masterpe But that did not solve it
14:16:21 masterpe is is active on compute48
14:17:24 sean-k-mooney you should not change that
14:17:46 sean-k-mooney actully that is strang
14:18:01 sean-k-mooney what hypervior are you using
14:19:25 masterpe qemu with kvm

Earlier   Later