Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-09-26
15:23:03 mnaser running 3 schedulers across 3 nodes was one big game of luck too
15:23:41 sean-k-mooney mnaser: when we stated clamin in placement you could jsut run multiple schuler process to scale without multiprocess support
15:25:52 jaypipes it was only the caching scheduler that was unsafe for multi-workers.
15:26:17 sean-k-mooney jaypipes: post placement
15:26:19 mriedem oh i'm not sure that's true
15:26:33 mriedem you could still race with filter scheduler
15:26:49 mriedem since there was no claim in the scheduler
15:27:00 sean-k-mooney mriedem: for numa/sriov resouces?
15:27:11 mriedem for anything
15:27:12 sean-k-mooney for a normal vm request i did not tihnk we coudl
15:27:13 mriedem before pike
15:27:51 sean-k-mooney mriedem: was pike when we started claiming cpus ram and disk in placmenent?
15:27:53 jaypipes right. before pike.
15:28:04 mriedem sean-k-mooney: yes
15:28:47 sean-k-mooney mriedem: ok so ya pre pike you could race post pike bar late claimes on the compute node for pinning/sriov/hugepages no race
15:30:58 mriedem or if you're using server groups...
15:31:02 openstackgerrit Artom Lifshitz proposed openstack/nova-specs master: Re-propose numa-aware-live-migration spec https://review.openstack.org/599587
15:31:20 mriedem so all of the NFV things = still race-tastic
15:31:55 mriedem and ironically the scheduler is the one thing that NFV things hate most because they already know exactly where they want to put the goodies
15:32:01 sean-k-mooney mriedem: thats true for multi process works and multi schduler too right they are both as rachy for nfv stuff
15:32:12 mriedem even single scheduler
15:32:14 mriedem single worker
15:32:51 mriedem this is why a late affinity check exists in the compute
15:39:24 mdbooth efried: Just wondering about external, non-libvirt CI in the context of https://review.openstack.org/#/c/602174/
15:40:12 mdbooth efried: Do you have one of those? If so, do you write your own jobs 'from scratch', or would the changes to the playbooks/.zuul.yaml in there cause it to fail?
15:41:31 efried mdbooth: We have an external, non-libvirt CI, yes. It runs tempest stuffs. I'm not an expert on it.
15:42:01 mdbooth efried: Ok. Just wondering if you'd end up running gate/test_evacuate.sh by default.
15:42:08 mdbooth If you did, it would fail.
15:42:24 mdbooth mriedem: Do you know how external CI is typically configured ^^^ ?
15:42:44 efried "typically"?
15:42:50 efried not really, no.
15:42:59 efried I couldn't really even tell you how ours is configured :)
15:43:09 efried edmondsw: Any ideas ^ ?
15:43:49 edmondsw not sure what gate/test_evacuate.sh is
15:43:51 openstackgerrit Matt Riedemann proposed openstack/nova master: Add attach kwarg to base/nova-net allocate_for_instance methods https://review.openstack.org/605464
15:43:56 jaypipes gibi: question for you on https://review.openstack.org/#/c/591811/. I'm sure I'm just missing something silly...
15:43:57 mriedem mdbooth: they wouldn't
15:43:59 efried mdbooth: It looks to me like this is only going to change the legacy nova-live-migration job. Does that even get triggered anymore?
15:44:06 mdbooth edmondsw: It's new in https://review.openstack.org/#/c/602174/
15:44:07 mriedem efried: yes
15:44:08 gibi jaypipes: looking
15:44:16 mriedem devstack-gate can run post-test hook scripts
15:44:39 mdbooth Basically if it's 'opt-in' then failing if libvirt isn't configured is the correct behaviour
15:44:40 mriedem some 3rd party CI still uses devstack-gate, some are moving to zuul v3 which doesn't use devstack-gate (unless you define a legacy-style job, like nova-live-migration)
15:44:48 mriedem it's definitely opt-in
15:44:53 mdbooth If we always run it we'd probably want it to just skip
15:44:59 mdbooth mriedem: Thanks
15:45:52 edmondsw mdbooth our CI is currently using devstack all-in-ones for each run, so it can't do things like evacuate that require multiple nodes yet... working on that
15:46:10 mriedem tempest doesn't test evacuate anyway
15:46:20 efried and we're using zuulv3, right?
15:46:23 mriedem that's why this in a separate script, when tempest isn't running
15:46:29 openstackgerrit Merged openstack/nova stable/queens: Follow devstack-plugin-ceph job rename https://review.openstack.org/602019
15:46:36 openstackgerrit Merged openstack/nova stable/queens: nova-status - don't count deleted compute_nodes https://review.openstack.org/604786
15:47:17 gibi jaypipes: you are right we are not catching AllocationDeleteFailed explicitly above in the call stack. Fortunately there are generic exception handling in place alreasy that puts the instance in ERROR state.
15:47:56 gibi jaypipes: the move operations are async on the API so when the fault happens there is no way to return that back to the API user anyhow
15:49:34 edmondsw efried we are not using zuulv3 in PowerVM CI yet, if that's what you were asking
15:49:51 efried dah, okay, thought we were
15:50:24 edmondsw last I heard, zuulv3 wasn't really ready for 3rd party CI usage yet
15:51:10 melwitt mriedem: I was looking at whether I should add the vmware live migration patch (in the queue) to a runway but saw it's failing vmware CI, and I see you've been discussing it with rado https://review.openstack.org/#/c/270116
15:53:05 mriedem i haven't looked at it since my last comments
15:53:48 mriedem it's also failing unit test
15:54:20 melwitt ok. I'll make a note next to it in the queue
15:57:42 openstack Launchpad bug 1789998 in OpenStack Compute (nova) "ResourceProviderAllocationRetrievalFailed ERROR log message on fresh n-cpu startup" [Low,Triaged]
15:57:42 mriedem a shiny donkey to whoever can bring me the head of https://bugs.launchpad.net/nova/+bug/1789998
15:58:13 efried F, I forgot *again* to collect my shiny nickel in Denver.
15:58:20 mriedem it's still in my backpack
15:58:53 efried That should probably be my bug. But I'm not likely to have time to look at it today.
15:59:55 efried also, /me wonders what "shiny donkey" means. Sounds like a euphemism for something.
16:00:00 efried Will it also fit in your backpack?
16:01:19 mdbooth mriedem: Speaking of common gate bugs: https://review.openstack.org/#/c/605436/
16:01:41 mriedem efried: https://www.youtube.com/watch?v=UNV44oqUF6k
16:01:56 mdbooth Although I didn't to a full test run on it locally first, so I won't be surprised if there's a kink to work out.
16:02:30 mriedem evacuate + affinity + locks = my head will explode
16:03:59 mdbooth mriedem: Add in a context manager which is a closure and some tail recursion ;)
16:05:13 melwitt I added cfriesen to the review
16:05:18 mdbooth cfriesen: https://review.openstack.org/#/c/605436/
16:05:32 mdbooth melwitt: Yeah, I was going to ping him earlier but he wasn't around
16:05:57 mdbooth I saw on the bug he looked at it before, and I assume there's some alternative solution in StarlingX
16:06:30 melwitt yeah
16:07:17 cfriesen for the "validate flavor extra-specs and image properties" work item, do we need a spec since it'll presumably result in an error message to the user? or are we allowed to return new error messages?
16:07:24 mdbooth Like I said it's not central to anything on my plate right now, though, so if somebody else wants to do a better job I'm cool with that. I probably won't spend a huge amount of time on it myself, though.
16:08:07 mdbooth I just fixed it because I saw it.
16:08:46 cfriesen mdbooth: taking a look
16:11:02 openstackgerrit Matthew Booth proposed openstack/nova master: Add volume-backed evacuate test https://review.openstack.org/604397
16:11:03 openstackgerrit Matthew Booth proposed openstack/nova master: Run evacuate tests with local/lvm and shared/rbd storage https://review.openstack.org/604400
16:11:04 cfriesen second question, for the "vcpu model extension" change where we'd allow specifying a list of CPU models in nova.conf instead of a single model, would we need a spec even though we're not changing the API?
16:11:32 mgariepy hello, I am upgrading from Pike to Queens but when running nova-manage db online_data_migrations, i get Some instances are still missing keypair information. Unable to run keypair migration at this time
16:14:13 mgariepy i found a few bug in lp concerning a workaround for kilo > liberty upgrade but the fix doesn't work for me as i don't have missing instance in my db.
16:14:16 openstack Launchpad bug 1684861 in OpenStack Compute (nova) newton "Mitaka -> Newton: Database online_data_migrations in newton fail due to missing keypairs" [Medium,In progress] - Assigned to Lee Yarwood (lyarwood)
16:14:16 mgariepy https://bugs.launchpad.net/nova/+bug/1684861
16:15:08 mgariepy I have 845 entry for select count(instance_uuid) from instance_extra where keypairs is NULL;
16:15:13 mdbooth mgariepy: See #topic. You should probably try #openstack instead
16:15:51 mgariepy well it's a nova issue.
16:16:12 mgariepy i've been upgrading to N o p q. and it fails a Q.
16:16:16 openstackgerrit Matthew Booth proposed openstack/nova master: Raise error on timeout in wait_for_versioned_notifications https://review.openstack.org/604859
16:16:17 openstackgerrit Matthew Booth proposed openstack/nova master: Don't delete disks on shared storage during evacuate https://review.openstack.org/578846
16:16:17 openstack bug 1550919 in OpenStack Compute (nova) "[Libvirt]Evacuate fail may cause disk image be deleted" [Medium,In progress] https://launchpad.net/bugs/1550919 - Assigned to Matthew Booth (mbooth-9)
16:16:17 openstackgerrit Matthew Booth proposed openstack/nova master: Add regression test for bug 1550919 https://review.openstack.org/591733
16:17:13 melwitt cfriesen: for extra spec and image properties validation, I think we would do a spec for it because it's an API change. for the cpu model list, from the ptg notes it looks like we thought we'd need a spec, maybe just to capture all of the related information. any opinion on either of these, mriedem?
16:17:31 mdbooth mgariepy: Indeed, but kilo and liberty are long out of support. Perhaps try your vendor?

Earlier   Later