Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-11-26
15:26:17 sean-k-mooney mriedem: well it either shouldn not exists or default to ture right
15:26:21 cdent mriedem: yes, I agree that it appears to be an anachronism that shouldn't be used at all, but that it sometimes does get used
15:26:33 mriedem i shall put out the dhellmann signal
15:26:33 cdent it is currently what makes my fix work
15:26:48 sean-k-mooney mriedem: there are some project that dont rung on python3 i belive
15:26:57 jmlowe Testing now, but it seems I can mitigate all of this with the timeout setting in the placement section of nova.conf
15:27:46 mriedem cdent: a peach?
15:28:27 mriedem jmlowe: do you have to set connection timeouts for other nova interactions, like with glance?
15:28:42 jmlowe I've never needed them before
15:29:39 mriedem weird. i think in queens we were going through ksa for glance interactions as well
15:29:45 mriedem so it should all be the same
15:29:54 mriedem from a client perspective i mean,
15:29:57 mriedem just different api servers
15:31:31 jmlowe I have been seeing the occasional ssl error in neutron clients, so could even be as simple as some sort of bug in haproxy
15:31:39 cfriesen mriedem: sean-k-mooney: yeah, I think you might be right, and most of the time it gets fixed up by the claim. ick.
15:33:23 sean-k-mooney cfriesen: mriedem we need to do a cleanup/audit of the migraion codepaths in general. artoms live migration spec will help but we likely have some cleanup to do for resize/cold migrate too.
15:34:07 cdent jmlowe: what's hosting your placement service? mod_wsgi, uwsgi, something else?
15:35:05 sean-k-mooney cfriesen: mriedem i know in pratcice cold migrate and resize "usually" work and results in the numa toplogy being recaluated but it may be just getting lucky or relying on retries. when we model numa in placemnt this will cange and we will have to correct the behavior for that
15:36:00 sean-k-mooney cfriesen: by the way i dont know if you saw my ML post regarding the pci numa affitiy policies
15:36:40 cdent efried: thanks for the +w on the external placement in nova change. I assume we want to let its child ( https://review.openstack.org/#/c/618215/ ) sit until more things have settled
15:36:43 cfriesen sean-k-mooney: glanced at it. haven't had a chance to dig in to it, they've got my focused on other stuff at the moment.
15:37:04 sean-k-mooney cfriesen: i did not test all the ploices but at least the prefer policy does not work as intended so we will need to fix that
15:37:14 efried cdent: Been avoiding that one until "WIP" goes away.
15:37:26 sean-k-mooney + add the ablity for it to work with neutron sriov ports
15:38:01 efried cdent: Basically taking any excuse to defer reviews while I try to get my feet back under me.
15:38:24 cdent efried: i've left it wip in expectation of it needing to hold. How do I mark something "i'd like reviews but this isn't done"? But yeah, understand the need to defer.
15:38:45 cdent (sometimes its the concept not the implementation that needs the review)
15:39:20 efried cdent: If you'd like me to -2 it, I could do that. Or you could -W it. Not sure either would help get it more attention, though.
15:39:28 mriedem cdent: should probably send a separate email to the ML about https://review.openstack.org/#/c/617941/ so nova people not paying much attention to placement know what's going on with tests now
15:39:30 efried what are we actually waiting on?
15:39:44 cdent mriedem: aye aye
15:40:02 mriedem b/c when i rebase and my tests start failing i'm going to wonder wtf
15:41:22 sean-k-mooney cdent: mriedem efried is https://review.openstack.org/#/c/599208/ still considerd a requiremetn before the placement extraction is complete
15:41:32 cdent yes
15:42:04 sean-k-mooney is there a list of what is out standing beyond that or is that the final feature
15:42:14 mriedem https://etherpad.openstack.org/p/BER-placement-extract
15:42:22 sean-k-mooney mriedem: cool thanks
15:43:06 mriedem bauzas: did you get a functional test written that does a reshape and then schedules to the child provider inventory resource?
15:44:32 mriedem cdent: why were these tests removed? https://review.openstack.org/#/c/617941/21/nova/tests/unit/cmd/test_status.py
15:44:35 mnaser https://review.openstack.org/#/c/619349/ simple backport if anyone has a second (i'll babysit the rest)
15:46:12 mriedem lyarwood: ^ just consider my backport a proxy +2 there
15:46:31 cdent mriedem: because the tests uses the rp_objects directly
15:46:48 cdent the follow on patch may even remove the status command since it can no longer work
15:47:08 cdent there's a fixme added in cmd/status.py
15:47:09 openstackgerrit Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459
15:47:10 openstackgerrit Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317
15:47:23 mriedem cdent: ok but the rp objects aren't removed in that patch, so it seemed out of place
15:47:30 mriedem we could count rps using the placement api right?
15:47:33 mriedem or using the fixture
15:48:37 cdent I think that one can still work, but not the inventory-related one
15:48:54 cdent I went through so many iterations on that series of changes, I may have lost my place
15:49:12 cdent I assumed then (and still do) that we'll need to do some "dynamic tidying"
15:49:48 mriedem on that size of change i can see why something would get lost
15:49:58 mriedem it just stuck out to me looking at it fresh
15:52:08 cdent my original plan was to not remove anything from nova in the first change, and just change the tests, but it proved to confusing why debugging.
15:52:10 mriedem anyway, i think the command could count compute resource providers by providers with VCPU inventory as it does today
15:52:16 mriedem and i know you have a spec for filtering that way as well
15:52:39 cdent I abandoned that spec because people weren't sure that was a sufficient use case, which confused me
15:52:55 mriedem heh, well we have a use case right here
15:53:41 mriedem nova-status upgrade check already hits the placement REST API to check the version in _check_placement
15:53:45 mriedem anywhere
15:53:47 mriedem *anyway
15:54:01 mriedem we might want to undo https://review.openstack.org/#/c/617941/21/nova/tests/unit/cmd/test_status.py in a follow up
15:54:20 mriedem unless we remove that upgrade check, but that's another discussion
15:54:55 cdent well, we'll have to at least change the upgrade check, in which case the test will have to change so it does rather go together
15:55:11 mriedem sure, the test has to exist to change it though
15:55:14 mriedem unless you're agreeing with me
15:55:39 cdent I'm mostly agreeing with you, except for the part about it being another discussion
15:55:49 cdent if we're going to remove the command, then job done
15:56:24 mriedem it depends on what nova-compute does on startup - if it can't create a resource provider b/c placement isn't running or nova-compute isn't configured to talk to it, then we can probably remove the check
15:56:44 mriedem otherwise it's useful as a base install verification that you've got computes reporting in and resource providers for those computes
15:57:46 mriedem FFU makes me nervous about when we can remove these things...
15:57:57 mriedem b/c anyone can FFU from any release to another presumably
15:58:32 mriedem having said that, i'm not sure these would work for FFU'ers anyway b/c the placement api would likely be down
15:58:53 mriedem dansmith: is that correct? we can expect placement-api to be down during an FFU?
16:00:24 sean-k-mooney cdent: placement does not randomise allocation candiates by defualt correct. they are just retruned in db order?
16:00:48 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#placement.randomize_allocation_candidates
16:02:53 sean-k-mooney mriedem: thanks ya its off by default. i was wondering if that could be related to this ml list post http://lists.openstack.org/pipermail/openstack-discuss/2018-November/000209.html
16:03:17 sean-k-mooney that said i woudl have expected the default weigher to kick in and spread the instances
16:04:08 cdent mriedem: yeah, I'm wondering if coupling nova-status to placement status is a good/safe idea? If they are supposed to be independently upgraded (for some value of "independent" maybe placement-status should do some kind of check? Except that we expect placement to upgrade first (usually). /me throws hands
16:05:02 cdent sean-k-mooney: that config item was left as not random by default so as to encoureage/allow pack, what would the weigher be doing?
16:05:15 cdent changing the config is certainly something they could at least try
16:05:32 sean-k-mooney cdent: the weighers used to spread by default
16:05:33 cdent my impression was that most of the system was biased towards packing to save $$
16:05:46 sean-k-mooney cdent: i think they still do
16:06:46 cdent probably still worth trying the config setting just to see?
16:06:51 mriedem cdent: i left a note to self / open question in the check code so i can maybe come back to it some other time when it comes up, or is a more pressing decision that needs to be made
16:07:09 sean-k-mooney ya i was also gong to ask them to provide the nova.conf the schduler is useing not the compute node one
16:07:23 cdent good point sean-k-mooney
16:07:27 cdent mriedem: ✔
16:07:55 dansmith mriedem: not just expect, but require
16:09:59 mriedem ok yeah then in that case, nova-status upgrade check will not work during FFU today
16:10:14 mriedem it will fail the check placement check since the API won't be up
16:10:48 dansmith I thought nova-status was to be db-only anyway? I guess not once placement is a different db, eh?
16:11:01 mriedem there is a check for the minimum placement version
16:11:10 mriedem otherwise it looks in the nova_api db yeah
16:11:31 dansmith so just have to make the api-bound check graceful I suppose
16:11:34 mriedem was discussing if we should change that to hit the placement API now, or just remove the placement-related checks
16:12:04 mriedem the upgrade checkers things were written before FFU was really a consideration i think
16:12:09 mriedem so i haven't put a ton of thought into it

Earlier   Later