Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-09-26
15:53:48 mriedem it's also failing unit test
15:54:20 melwitt ok. I'll make a note next to it in the queue
15:57:42 openstack Launchpad bug 1789998 in OpenStack Compute (nova) "ResourceProviderAllocationRetrievalFailed ERROR log message on fresh n-cpu startup" [Low,Triaged]
15:57:42 mriedem a shiny donkey to whoever can bring me the head of https://bugs.launchpad.net/nova/+bug/1789998
15:58:13 efried F, I forgot *again* to collect my shiny nickel in Denver.
15:58:20 mriedem it's still in my backpack
15:58:53 efried That should probably be my bug. But I'm not likely to have time to look at it today.
15:59:55 efried also, /me wonders what "shiny donkey" means. Sounds like a euphemism for something.
16:00:00 efried Will it also fit in your backpack?
16:01:19 mdbooth mriedem: Speaking of common gate bugs: https://review.openstack.org/#/c/605436/
16:01:41 mriedem efried: https://www.youtube.com/watch?v=UNV44oqUF6k
16:01:56 mdbooth Although I didn't to a full test run on it locally first, so I won't be surprised if there's a kink to work out.
16:02:30 mriedem evacuate + affinity + locks = my head will explode
16:03:59 mdbooth mriedem: Add in a context manager which is a closure and some tail recursion ;)
16:05:13 melwitt I added cfriesen to the review
16:05:18 mdbooth cfriesen: https://review.openstack.org/#/c/605436/
16:05:32 mdbooth melwitt: Yeah, I was going to ping him earlier but he wasn't around
16:05:57 mdbooth I saw on the bug he looked at it before, and I assume there's some alternative solution in StarlingX
16:06:30 melwitt yeah
16:07:17 cfriesen for the "validate flavor extra-specs and image properties" work item, do we need a spec since it'll presumably result in an error message to the user? or are we allowed to return new error messages?
16:07:24 mdbooth Like I said it's not central to anything on my plate right now, though, so if somebody else wants to do a better job I'm cool with that. I probably won't spend a huge amount of time on it myself, though.
16:08:07 mdbooth I just fixed it because I saw it.
16:08:46 cfriesen mdbooth: taking a look
16:11:02 openstackgerrit Matthew Booth proposed openstack/nova master: Add volume-backed evacuate test https://review.openstack.org/604397
16:11:03 openstackgerrit Matthew Booth proposed openstack/nova master: Run evacuate tests with local/lvm and shared/rbd storage https://review.openstack.org/604400
16:11:04 cfriesen second question, for the "vcpu model extension" change where we'd allow specifying a list of CPU models in nova.conf instead of a single model, would we need a spec even though we're not changing the API?
16:11:32 mgariepy hello, I am upgrading from Pike to Queens but when running nova-manage db online_data_migrations, i get Some instances are still missing keypair information. Unable to run keypair migration at this time
16:14:13 mgariepy i found a few bug in lp concerning a workaround for kilo > liberty upgrade but the fix doesn't work for me as i don't have missing instance in my db.
16:14:16 openstack Launchpad bug 1684861 in OpenStack Compute (nova) newton "Mitaka -> Newton: Database online_data_migrations in newton fail due to missing keypairs" [Medium,In progress] - Assigned to Lee Yarwood (lyarwood)
16:14:16 mgariepy https://bugs.launchpad.net/nova/+bug/1684861
16:15:08 mgariepy I have 845 entry for select count(instance_uuid) from instance_extra where keypairs is NULL;
16:15:13 mdbooth mgariepy: See #topic. You should probably try #openstack instead
16:15:51 mgariepy well it's a nova issue.
16:16:12 mgariepy i've been upgrading to N o p q. and it fails a Q.
16:16:16 openstackgerrit Matthew Booth proposed openstack/nova master: Raise error on timeout in wait_for_versioned_notifications https://review.openstack.org/604859
16:16:17 openstackgerrit Matthew Booth proposed openstack/nova master: Don't delete disks on shared storage during evacuate https://review.openstack.org/578846
16:16:17 openstack bug 1550919 in OpenStack Compute (nova) "[Libvirt]Evacuate fail may cause disk image be deleted" [Medium,In progress] https://launchpad.net/bugs/1550919 - Assigned to Matthew Booth (mbooth-9)
16:16:17 openstackgerrit Matthew Booth proposed openstack/nova master: Add regression test for bug 1550919 https://review.openstack.org/591733
16:17:13 melwitt cfriesen: for extra spec and image properties validation, I think we would do a spec for it because it's an API change. for the cpu model list, from the ptg notes it looks like we thought we'd need a spec, maybe just to capture all of the related information. any opinion on either of these, mriedem?
16:17:31 mdbooth mgariepy: Indeed, but kilo and liberty are long out of support. Perhaps try your vendor?
16:18:42 mdbooth mriedem: We don't have any NFS CI jobs, do we?
16:18:44 mgariepy i'm upgrading from pike to queens.
16:19:11 mdbooth mriedem: I probably asked this before: my memory is terrible.
16:23:38 imacdonn mgariepy: you should at least try in #openstack ... "how do I...?" questions should start there. This channel is about development, not deployment .. if it's determined that there's a current bug, it could be brought here
16:24:34 mriedem mdbooth: we do,
16:24:37 mriedem it's in the experimental queue
16:24:47 mriedem melwitt: yes for api spec for extra spec validation in the api
16:24:59 mriedem as for cpu models stuff in nova.conf, idk, wasn't paying attention to that at the ptg
16:25:35 mriedem mdbooth: pro tip: look in nova's .zuul.yaml file
16:25:46 mriedem ye shall behold legacy-tempest-dsvm-full-devstack-plugin-nfs
16:26:01 cfriesen mdbooth: the starlingx server group validation stuff is here: https://github.com/starlingx-staging/stx-nova/blob/master/nova/compute/manager.py#L1408-L1436 and the check against "older" instances is here: https://github.com/starlingx-staging/stx-nova/blob/master/nova/objects/instance_group.py#L550-L572
16:26:30 cfriesen melwitt: okay, specs it is.
16:26:36 mriedem cfriesen: i think i might have mentioned this to you before, but you know how starlingx has a patched/upgraded flag it sets on the HostState object in the scheduler and then has a weigher for those?
16:26:45 melwitt thanks. cfriesen ^ you could try the cpu models as a specless bp and when we ask for approval during the nova meeting, someone might point out why it should be a spec, so you might have to write one at that point
16:26:49 mriedem i think the idea being, send new requests to patched/upgraded hosts?
16:27:19 mriedem cfriesen: any reason to not just check the compute's service version to see if it's the latest?
16:27:22 mriedem that would tell you if it's upgraded
16:28:11 melwitt mgariepy: you said earlier that you have not manually deleted any instances from the database?
16:28:14 cfriesen mriedem: one issue was that patching for a bugfix might not affect the service version
16:28:28 cfriesen mriedem: but that would work for the upgrade case
16:29:02 mgariepy melwitt, nop i didn't
16:29:11 mriedem weighing based on bug fix patches seems excessive
16:29:26 mgariepy i think the upgrade db didn't updated the deleted instances attrbutes.
16:29:35 mriedem but i guess i get it
16:30:32 cfriesen mriedem: so in the original model due to limitations we had to reboot the compute node when patching, so when rolling out a patch we really didn't want to have to migrate instances multiple times if we could avoid it. Probably less of an issue now.
16:31:16 mgariepy melwitt, updated the keypairs fields in the db..
16:34:33 imacdonn efried: ping me if you want to discuss https://review.openstack.org/#/c/605329/ - there's probably a sexier way to do it
16:35:15 mriedem imacdonn: gonna need tests
16:35:32 mriedem b/c clearly we weren't testing this before which is why it's a bug
16:35:41 imacdonn mriedem: yeah, I figured that'd come up ... but there aren't any tests at all for this stuff currently, or I failed to find them
16:36:28 imacdonn Trying to verify the contents of a PrettyTable didn't seem appealing :/
16:36:39 mriedem https://github.com/openstack/nova/blob/41ac87f812f239e38ad3b72c2f2108523ad3444d/nova/tests/unit/test_nova_manage.py#L749
16:37:08 mriedem there are tests for each individual migration method but you're right that they aren't tested together in aggregate
16:37:09 imacdonn oh. Why isn't that under "cmd" ?
16:37:16 mriedem b/c history
16:37:19 imacdonn mmkay
16:37:28 mriedem no reason we couldn't move it
16:37:35 mriedem except it makes backports a pain in the ass
16:37:37 melwitt mgariepy: oh, hm. ok, we have had online data migrations before that skip deleted records, so that sounds legit but I need to look and double check. it's surprising though that we haven't had this reported before, I'd think it would be common
16:38:10 imacdonn k, I'll look into why the test didn't catch the bug
16:38:40 imacdonn oh, I guess because it didn't have more than 50 of any migration type
16:40:44 melwitt tssurya, mriedem: I've got a call at the top of the hour, so I won't be able to participate much in a cells meeting if we have one
16:43:21 mgariepy melwitt, the cloud i'm upgarding lives from some times. first install was kilo and upgraded
16:43:43 mgariepy but depending on the when it's upgraded, it can cause inconsistency some times..
16:46:07 melwitt mgariepy: yeah, so I'm looking at this and the migration is that it's moving keypairs from the "main" database to the api database. and it's counting only the non-deleted InstanceExtra. InstanceExtra are marked as deleted when instances are deleted through the nova API https://github.com/openstack/nova/blob/stable/queens/nova/objects/keypair.py#L245
16:50:20 melwitt I don't yet understand how this works, I see the migration creates a keypair in the api db, deletes the keypair from the cell db, but I'm not yet seeing the connection with instance_extra
16:51:48 melwitt it looks like the instance_extra part is a separate migration that should have happened before this cell db => api db one that I'm looking at
16:52:43 melwitt ok yeah, instance_obj.migrate_instance_keypairs
16:52:49 melwitt from newton
16:53:21 mgariepy i'm currently using : nova_git_install_branch: d7864fbb9c2c558c409559e1d5989f84c7403832 # HEAD of "stable/queens" as of 23.07.2018
16:53:54 mgariepy melwitt, can you point me to the commit ?
16:54:51 melwitt mgariepy: here's the code https://github.com/openstack/nova/blob/stable/ocata/nova/objects/instance.py#L1398 that should have run when you upgraded to newton, when you ran the nova-manage db online_data_migrations. did you do that for newton?
16:55:53 melwitt the code no longer exists in queens, so if you missed running that online data migration in newton, you can't run it with the queens code
16:56:16 mgariepy i ran it for sure.
16:58:33 melwitt and there were no errors? because while this is only migrating non-deleted records, the check that is blocking you in queens is also only considering non-deleted records, so I don't see how this could be happening unless the online migration from newton was missed or had errors
16:58:53 openstackgerrit Merged openstack/nova master: placement: Always reset conf.CONF when starting the wsgi app https://review.openstack.org/604693
17:00:24 melwitt as for getting out of it, if the unmigrated instance_extra records are associated with non-deleted instances, I think you're going to have to do something like use the old newton code to migrate the records. if they unmigrated instance_extra records are associated with deleted instances, you're going to have to delete them manually
17:02:27 mgariepy the Ocata code i was runnig wasn't the latest
17:03:16 mgariepy anyway seems to be fixed now.
17:04:34 mgariepy is there a script to validate the database ?

Earlier   Later